[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781935603120653-11672-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 06:06:43 dist-test-slave-9hm5 krb5kdc[11679](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 06:06:43 dist-test-slave-9hm5 krb5kdc[11679](info): set up 2 sockets
Jun 20 06:06:43 dist-test-slave-9hm5 krb5kdc[11679](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260620 06:06:44.188202 11672 mini_kdc.cc:121] Time spent starting KDC: real 1.032s user 0.005s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781935603120653-11672-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781935603120653-11672-0/krb5kdc/test-user.keytab.
Jun 20 06:06:44 dist-test-slave-9hm5 krb5kdc[11679](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935604, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T06:06:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T06:06:44Z Disabled control of system clock
WARNING: no policy specified for kudu/127.11.102.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.102.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.102.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781935603120653-11672-0/krb5kdc/kudu_127.11.102.62.keytab.
Entry for principal kudu/127.11.102.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781935603120653-11672-0/krb5kdc/kudu_127.11.102.62.keytab.
I20260620 06:06:44.306569 11672 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9HMp43/build/debug/bin/kudu
/tmp/dist-test-task9HMp43/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781935603120653-11672-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781935603120653-11672-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781935603120653-11672-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781935603120653-11672-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.102.62:44285
--webserver_interface=127.11.102.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781935603120653-11672-0/krb5kdc/kudu_127.11.102.62.keytab
--principal=kudu/127.11.102.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.102.20:43587
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.102.62:44285
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781935603120653-11672-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781935603120653-11672-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781935603120653-11672-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 06:06:44.413520 11692 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:06:44.413832 11692 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:06:44.413920 11692 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:06:44.417986 11692 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 06:06:44.418073 11692 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:06:44.418111 11692 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:06:44.418169 11692 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 06:06:44.418208 11692 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 06:06:44.423344 11692 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.102.20:43587
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781935603120653-11672-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781935603120653-11672-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.102.62:44285
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781935603120653-11672-0/krb5kdc/kudu_127.11.102.62.keytab
--principal=kudu/127.11.102.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.102.62:44285
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781935603120653-11672-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.102.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11692
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781935603120653-11672-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 05:28:48 UTC on ccycloud.quasar-fixhms.root.comops.site
build id 4457020
I20260620 06:06:44.424392 11692 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:06:44.425427 11692 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 06:06:44.434149 11697 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:06:44.434140 11698 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:06:44.434252 11692 server_base.cc:1034] running on GCE node
W20260620 06:06:44.434260 11700 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:06:44.434952 11692 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:06:44.436098 11692 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:06:44.437284 11692 hybrid_clock.cc:648] HybridClock initialized: now 1781935604437216 us; error 81 us; skew 500 ppm
Jun 20 06:06:44 dist-test-slave-9hm5 krb5kdc[11679](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935604, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.102.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 06:06:44.440024 11692 init.cc:377] Logged in from keytab as kudu/127.11.102.62@KRBTEST.COM (short username kudu)
I20260620 06:06:44.440856 11692 webserver.cc:460] Webserver started at http://127.11.102.62:41471/ using document root <none> and password file <none>
I20260620 06:06:44.441407 11692 fs_manager.cc:359] Metadata directory not provided
I20260620 06:06:44.441505 11692 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:06:44.441731 11692 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 06:06:44.443773 11692 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781935603120653-11672-0/minicluster-data/master-0/data/instance:
uuid: "1eb71ab511f248d187826ff534455fe2"
format_stamp: "Formatted at 2026-06-20 06:06:44 on dist-test-slave-9hm5"
server_key: "3f3cc2d971c7b4ecc69cdeacae2fd5f4"
server_key_iv: "370a4d7b777ae21583ced8e3d35dcd8d"
server_key_version: "encryptionkey@0"
I20260620 06:06:44.444336 11692 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781935603120653-11672-0/minicluster-data/master-0/wal/instance:
uuid: "1eb71ab511f248d187826ff534455fe2"
format_stamp: "Formatted at 2026-06-20 06:06:44 on dist-test-slave-9hm5"
server_key: "3f3cc2d971c7b4ecc69cdeacae2fd5f4"
server_key_iv: "370a4d7b777ae21583ced8e3d35dcd8d"
server_key_version: "encryptionkey@0"
I20260620 06:06:44.447942 11692 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.002s
I20260620 06:06:44.450320 11707 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:06:44.451558 11692 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 06:06:44.451748 11692 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781935603120653-11672-0/minicluster-data/master-0/data,/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781935603120653-11672-0/minicluster-data/master-0/wal
uuid: "1eb71ab511f248d187826ff534455fe2"
format_stamp: "Formatted at 2026-06-20 06:06:44 on dist-test-slave-9hm5"
server_key: "3f3cc2d971c7b4ecc69cdeacae2fd5f4"
server_key_iv: "370a4d7b777ae21583ced8e3d35dcd8d"
server_key_version: "encryptionkey@0"
I20260620 06:06:44.451874 11692 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781935603120653-11672-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781935603120653-11672-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781935603120653-11672-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:06:44.481184 11692 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:06:44.484896 11692 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:06:44.485123 11692 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:06:44.492795 11692 rpc_server.cc:307] RPC server started. Bound to: 127.11.102.62:44285
I20260620 06:06:44.492825 11759 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.102.62:44285 every 8 connection(s)
I20260620 06:06:44.493786 11692 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781935603120653-11672-0/minicluster-data/master-0/data/info.pb
I20260620 06:06:44.496547 11760 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 06:06:44.501915 11760 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1eb71ab511f248d187826ff534455fe2: Bootstrap starting.
I20260620 06:06:44.503139 11672 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9HMp43/build/debug/bin/kudu as pid 11692
I20260620 06:06:44.503309 11672 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781935603120653-11672-0/minicluster-data/master-0/wal/instance
I20260620 06:06:44.504926 11672 external_mini_cluster.cc:1428] Setting key 1516e8f35bed9ec6ecb6f4868405ffde
I20260620 06:06:44.504952 11760 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1eb71ab511f248d187826ff534455fe2: Neither blocks nor log segments found. Creating new log.
I20260620 06:06:44.505795 11760 log.cc:824] T 00000000000000000000000000000000 P 1eb71ab511f248d187826ff534455fe2: Log is configured to *not* fsync() on all Append() calls
I20260620 06:06:44.507952 11760 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1eb71ab511f248d187826ff534455fe2: No bootstrap required, opened a new log
I20260620 06:06:44.512010 11760 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1eb71ab511f248d187826ff534455fe2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1eb71ab511f248d187826ff534455fe2" member_type: VOTER last_known_addr { host: "127.11.102.62" port: 44285 } }
I20260620 06:06:44.512248 11760 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1eb71ab511f248d187826ff534455fe2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 06:06:44.512328 11760 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1eb71ab511f248d187826ff534455fe2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1eb71ab511f248d187826ff534455fe2, State: Initialized, Role: FOLLOWER
I20260620 06:06:44.512806 11760 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1eb71ab511f248d187826ff534455fe2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1eb71ab511f248d187826ff534455fe2" member_type: VOTER last_known_addr { host: "127.11.102.62" port: 44285 } }
I20260620 06:06:44.512943 11760 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1eb71ab511f248d187826ff534455fe2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 06:06:44.513010 11760 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1eb71ab511f248d187826ff534455fe2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 06:06:44.513095 11760 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1eb71ab511f248d187826ff534455fe2 [term 0 FOLLOWER]: Advancing to term 1
I20260620 06:06:44.514149 11760 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1eb71ab511f248d187826ff534455fe2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1eb71ab511f248d187826ff534455fe2" member_type: VOTER last_known_addr { host: "127.11.102.62" port: 44285 } }
I20260620 06:06:44.514470 11760 leader_election.cc:302] T 00000000000000000000000000000000 P 1eb71ab511f248d187826ff534455fe2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1eb71ab511f248d187826ff534455fe2; no voters:
I20260620 06:06:44.514778 11760 leader_election.cc:288] T 00000000000000000000000000000000 P 1eb71ab511f248d187826ff534455fe2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 06:06:44.514839 11765 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1eb71ab511f248d187826ff534455fe2 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 06:06:44.515050 11765 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1eb71ab511f248d187826ff534455fe2 [term 1 LEADER]: Becoming Leader. State: Replica: 1eb71ab511f248d187826ff534455fe2, State: Running, Role: LEADER
I20260620 06:06:44.515326 11765 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1eb71ab511f248d187826ff534455fe2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1eb71ab511f248d187826ff534455fe2" member_type: VOTER last_known_addr { host: "127.11.102.62" port: 44285 } }
I20260620 06:06:44.516065 11760 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1eb71ab511f248d187826ff534455fe2 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 06:06:44.516968 11767 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1eb71ab511f248d187826ff534455fe2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1eb71ab511f248d187826ff534455fe2. Latest consensus state: current_term: 1 leader_uuid: "1eb71ab511f248d187826ff534455fe2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1eb71ab511f248d187826ff534455fe2" member_type: VOTER last_known_addr { host: "127.11.102.62" port: 44285 } } }
I20260620 06:06:44.517125 11767 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1eb71ab511f248d187826ff534455fe2 [sys.catalog]: This master's current role is: LEADER
I20260620 06:06:44.517438 11766 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1eb71ab511f248d187826ff534455fe2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1eb71ab511f248d187826ff534455fe2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1eb71ab511f248d187826ff534455fe2" member_type: VOTER last_known_addr { host: "127.11.102.62" port: 44285 } } }
I20260620 06:06:44.517549 11766 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1eb71ab511f248d187826ff534455fe2 [sys.catalog]: This master's current role is: LEADER
I20260620 06:06:44.517819 11774 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 06:06:44.520946 11774 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Jun 20 06:06:44 dist-test-slave-9hm5 krb5kdc[11679](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935604, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.102.62@KRBTEST.COM
I20260620 06:06:44.528641 11774 catalog_manager.cc:1347] Generated new cluster ID: 757117c71ada4536a113206a629a7cd4
I20260620 06:06:44.528707 11774 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 06:06:44.531068 11763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:06:44.508330 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57020 (local address 127.11.102.62:44285)
0620 06:06:44.508765 (+ 435us) server_negotiation.cc:206] Beginning negotiation
0620 06:06:44.508784 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0620 06:06:44.509960 (+ 1176us) server_negotiation.cc:407] Connection header received
0620 06:06:44.512658 (+ 2698us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:06:44.512677 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:06:44.513050 (+ 373us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:06:44.513416 (+ 366us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:06:44.516061 (+ 2645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:06:44.517614 (+ 1553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:06:44.519395 (+ 1781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:06:44.519964 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:06:44.520149 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:06:44.525021 (+ 4872us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:06:44.525028 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:06:44.525043 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:06:44.525077 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:06:44.527239 (+ 2162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:06:44.527946 (+ 707us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:06:44.527952 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:06:44.527956 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:06:44.528110 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:06:44.528520 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:06:44.528523 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:06:44.528525 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:06:44.529066 (+ 541us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:06:44.529291 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0620 06:06:44.529836 (+ 545us) server_negotiation.cc:299] Negotiation successful
0620 06:06:44.530146 (+ 310us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":276,"thread_start_us":113,"threads_started":1}
I20260620 06:06:44.574853 11774 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 06:06:44.576179 11774 catalog_manager.cc:1504] Loading token signing keys...
I20260620 06:06:44.580928 11774 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1eb71ab511f248d187826ff534455fe2: Generated new TSK 0
I20260620 06:06:44.581744 11774 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.11.102.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.102.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.102.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781935603120653-11672-0/krb5kdc/kudu_127.11.102.1.keytab.
Entry for principal kudu/127.11.102.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781935603120653-11672-0/krb5kdc/kudu_127.11.102.1.keytab.
I20260620 06:06:44.617226 11672 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9HMp43/build/debug/bin/kudu
/tmp/dist-test-task9HMp43/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781935603120653-11672-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781935603120653-11672-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781935603120653-11672-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781935603120653-11672-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.102.1:0
--local_ip_for_outbound_sockets=127.11.102.1
--webserver_interface=127.11.102.1
--webserver_port=0
--tserver_master_addrs=127.11.102.62:44285
--keytab_file=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781935603120653-11672-0/krb5kdc/kudu_127.11.102.1.keytab
--principal=kudu/127.11.102.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.102.20:43587
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781935603120653-11672-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781935603120653-11672-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781935603120653-11672-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 06:06:44.725723 11786 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:06:44.725993 11786 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:06:44.726075 11786 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:06:44.730062 11786 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:06:44.730147 11786 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:06:44.730239 11786 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.102.1
I20260620 06:06:44.735240 11786 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.102.20:43587
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781935603120653-11672-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781935603120653-11672-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781935603120653-11672-0/krb5kdc/kudu_127.11.102.1.keytab
--principal=kudu/127.11.102.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.102.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781935603120653-11672-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.11.102.1
--webserver_port=0
--tserver_master_addrs=127.11.102.62:44285
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11786
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.102.1
--log_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781935603120653-11672-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 05:28:48 UTC on ccycloud.quasar-fixhms.root.comops.site
build id 4457020
I20260620 06:06:44.736316 11786 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:06:44.737309 11786 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 06:06:44.746781 11786 server_base.cc:1034] running on GCE node
W20260620 06:06:44.746716 11792 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:06:44.746716 11791 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:06:44.746716 11794 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:06:44.747459 11786 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:06:44.748117 11786 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:06:44.749313 11786 hybrid_clock.cc:648] HybridClock initialized: now 1781935604749297 us; error 30 us; skew 500 ppm
Jun 20 06:06:44 dist-test-slave-9hm5 krb5kdc[11679](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935604, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.102.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 06:06:44.752048 11786 init.cc:377] Logged in from keytab as kudu/127.11.102.1@KRBTEST.COM (short username kudu)
I20260620 06:06:44.752921 11786 webserver.cc:460] Webserver started at http://127.11.102.1:39379/ using document root <none> and password file <none>
I20260620 06:06:44.753497 11786 fs_manager.cc:359] Metadata directory not provided
I20260620 06:06:44.753600 11786 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:06:44.753804 11786 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 06:06:44.755937 11786 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781935603120653-11672-0/minicluster-data/ts-0/data/instance:
uuid: "f0734b581b48497eba3229a2f15a892f"
format_stamp: "Formatted at 2026-06-20 06:06:44 on dist-test-slave-9hm5"
server_key: "5bcb8b66175148bf3cce8c7d9939c998"
server_key_iv: "9b3b89e9c444d9b8be151ae54efec9eb"
server_key_version: "encryptionkey@0"
I20260620 06:06:44.756522 11786 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781935603120653-11672-0/minicluster-data/ts-0/wal/instance:
uuid: "f0734b581b48497eba3229a2f15a892f"
format_stamp: "Formatted at 2026-06-20 06:06:44 on dist-test-slave-9hm5"
server_key: "5bcb8b66175148bf3cce8c7d9939c998"
server_key_iv: "9b3b89e9c444d9b8be151ae54efec9eb"
server_key_version: "encryptionkey@0"
I20260620 06:06:44.760028 11786 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260620 06:06:44.762247 11801 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:06:44.763454 11786 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 06:06:44.763593 11786 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781935603120653-11672-0/minicluster-data/ts-0/data,/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781935603120653-11672-0/minicluster-data/ts-0/wal
uuid: "f0734b581b48497eba3229a2f15a892f"
format_stamp: "Formatted at 2026-06-20 06:06:44 on dist-test-slave-9hm5"
server_key: "5bcb8b66175148bf3cce8c7d9939c998"
server_key_iv: "9b3b89e9c444d9b8be151ae54efec9eb"
server_key_version: "encryptionkey@0"
I20260620 06:06:44.763746 11786 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781935603120653-11672-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781935603120653-11672-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781935603120653-11672-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:06:44.778265 11786 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:06:44.781325 11786 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:06:44.781528 11786 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:06:44.782094 11786 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 06:06:44.782928 11786 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 06:06:44.783001 11786 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 06:06:44.783068 11786 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 06:06:44.783113 11786 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 06:06:44.792112 11786 rpc_server.cc:307] RPC server started. Bound to: 127.11.102.1:44961
I20260620 06:06:44.792137 11914 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.102.1:44961 every 8 connection(s)
I20260620 06:06:44.792948 11786 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781935603120653-11672-0/minicluster-data/ts-0/data/info.pb
I20260620 06:06:44.793500 11672 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9HMp43/build/debug/bin/kudu as pid 11786
I20260620 06:06:44.793599 11672 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781935603120653-11672-0/minicluster-data/ts-0/wal/instance
I20260620 06:06:44.793848 11672 external_mini_cluster.cc:1428] Setting key 71e1a14c3d7b629516e4a657b313e3b2
WARNING: no policy specified for kudu/127.11.102.2@KRBTEST.COM; defaulting to no policy
Jun 20 06:06:44 dist-test-slave-9hm5 krb5kdc[11679](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935604, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.102.1@KRBTEST.COM for kudu/127.11.102.62@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.102.2@KRBTEST.COM" created.
I20260620 06:06:44.808328 11763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:06:44.794643 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.102.1:35133 (local address 127.11.102.62:44285)
0620 06:06:44.794776 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0620 06:06:44.794779 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 06:06:44.796022 (+ 1243us) server_negotiation.cc:407] Connection header received
0620 06:06:44.797357 (+ 1335us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:06:44.797362 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:06:44.797420 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:06:44.797532 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:06:44.798692 (+ 1160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:06:44.799605 (+ 913us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:06:44.801164 (+ 1559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:06:44.801579 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:06:44.801695 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:06:44.804688 (+ 2993us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:06:44.804691 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:06:44.804693 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:06:44.804715 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:06:44.806269 (+ 1554us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:06:44.806771 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:06:44.806776 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:06:44.806778 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:06:44.806861 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:06:44.807195 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:06:44.807198 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:06:44.807200 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:06:44.807375 (+ 175us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:06:44.807482 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0620 06:06:44.808038 (+ 556us) server_negotiation.cc:299] Negotiation successful
0620 06:06:44.808156 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260620 06:06:44.809345 11917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:06:44.795143 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.102.62:44285 (local address 127.11.102.1:35133)
0620 06:06:44.795866 (+ 723us) negotiation.cc:107] Waiting for socket to connect
0620 06:06:44.795920 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0620 06:06:44.797131 (+ 1211us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:06:44.797675 (+ 544us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:06:44.797681 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:06:44.797998 (+ 317us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:06:44.798548 (+ 550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:06:44.798561 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:06:44.799741 (+ 1180us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:06:44.799743 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:06:44.801026 (+ 1283us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:06:44.801037 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:06:44.801871 (+ 834us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:06:44.801875 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:06:44.802079 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:06:44.802683 (+ 604us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:06:44.802708 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:06:44.804540 (+ 1832us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:06:44.806397 (+ 1857us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:06:44.806402 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:06:44.806416 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:06:44.806669 (+ 253us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:06:44.806952 (+ 283us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:06:44.806957 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:06:44.806959 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:06:44.807093 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:06:44.807478 (+ 385us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:06:44.807484 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:06:44.807797 (+ 313us) client_negotiation.cc:769] Sending connection context
0620 06:06:44.807977 (+ 180us) client_negotiation.cc:240] Negotiation successful
0620 06:06:44.808276 (+ 299us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":504,"thread_start_us":106,"threads_started":1}
I20260620 06:06:44.810631 11915 heartbeater.cc:347] Connected to a master server at 127.11.102.62:44285
I20260620 06:06:44.810875 11915 heartbeater.cc:464] Registering TS with master...
I20260620 06:06:44.811328 11915 heartbeater.cc:511] Master 127.11.102.62:44285 requested a full tablet report, sending...
I20260620 06:06:44.812803 11724 ts_manager.cc:194] Registered new tserver with Master: f0734b581b48497eba3229a2f15a892f (127.11.102.1:44961)
I20260620 06:06:44.813872 11724 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.102.1@KRBTEST.COM'} at 127.11.102.1:35133
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.102.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781935603120653-11672-0/krb5kdc/kudu_127.11.102.2.keytab.
Entry for principal kudu/127.11.102.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781935603120653-11672-0/krb5kdc/kudu_127.11.102.2.keytab.
I20260620 06:06:44.821763 11672 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9HMp43/build/debug/bin/kudu
/tmp/dist-test-task9HMp43/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781935603120653-11672-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781935603120653-11672-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781935603120653-11672-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781935603120653-11672-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.102.2:0
--local_ip_for_outbound_sockets=127.11.102.2
--webserver_interface=127.11.102.2
--webserver_port=0
--tserver_master_addrs=127.11.102.62:44285
--keytab_file=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781935603120653-11672-0/krb5kdc/kudu_127.11.102.2.keytab
--principal=kudu/127.11.102.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.102.20:43587
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781935603120653-11672-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781935603120653-11672-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781935603120653-11672-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 06:06:44.931242 11920 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:06:44.931488 11920 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:06:44.931530 11920 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:06:44.935451 11920 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:06:44.935515 11920 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:06:44.935573 11920 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.102.2
I20260620 06:06:44.940649 11920 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.102.20:43587
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781935603120653-11672-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781935603120653-11672-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781935603120653-11672-0/krb5kdc/kudu_127.11.102.2.keytab
--principal=kudu/127.11.102.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.102.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781935603120653-11672-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.11.102.2
--webserver_port=0
--tserver_master_addrs=127.11.102.62:44285
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11920
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.102.2
--log_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781935603120653-11672-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 05:28:48 UTC on ccycloud.quasar-fixhms.root.comops.site
build id 4457020
I20260620 06:06:44.941686 11920 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:06:44.942709 11920 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 06:06:44.951983 11926 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:06:44.952049 11920 server_base.cc:1034] running on GCE node
W20260620 06:06:44.951967 11925 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:06:44.951963 11928 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:06:44.952634 11920 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:06:44.953233 11920 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:06:44.954399 11920 hybrid_clock.cc:648] HybridClock initialized: now 1781935604954383 us; error 25 us; skew 500 ppm
Jun 20 06:06:44 dist-test-slave-9hm5 krb5kdc[11679](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935604, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.102.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 06:06:44.957612 11920 init.cc:377] Logged in from keytab as kudu/127.11.102.2@KRBTEST.COM (short username kudu)
I20260620 06:06:44.958421 11920 webserver.cc:460] Webserver started at http://127.11.102.2:40571/ using document root <none> and password file <none>
I20260620 06:06:44.958954 11920 fs_manager.cc:359] Metadata directory not provided
I20260620 06:06:44.959015 11920 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:06:44.959218 11920 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 06:06:44.961225 11920 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781935603120653-11672-0/minicluster-data/ts-1/data/instance:
uuid: "9cd67b8a38e34e1985d33c4bfbb497eb"
format_stamp: "Formatted at 2026-06-20 06:06:44 on dist-test-slave-9hm5"
server_key: "5ad2101622c85148fbccfc81777fc00c"
server_key_iv: "39aa9575892624135a3a9df1da459efa"
server_key_version: "encryptionkey@0"
I20260620 06:06:44.961776 11920 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781935603120653-11672-0/minicluster-data/ts-1/wal/instance:
uuid: "9cd67b8a38e34e1985d33c4bfbb497eb"
format_stamp: "Formatted at 2026-06-20 06:06:44 on dist-test-slave-9hm5"
server_key: "5ad2101622c85148fbccfc81777fc00c"
server_key_iv: "39aa9575892624135a3a9df1da459efa"
server_key_version: "encryptionkey@0"
I20260620 06:06:44.965345 11920 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260620 06:06:44.968142 11935 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:06:44.969323 11920 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 06:06:44.969439 11920 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781935603120653-11672-0/minicluster-data/ts-1/data,/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781935603120653-11672-0/minicluster-data/ts-1/wal
uuid: "9cd67b8a38e34e1985d33c4bfbb497eb"
format_stamp: "Formatted at 2026-06-20 06:06:44 on dist-test-slave-9hm5"
server_key: "5ad2101622c85148fbccfc81777fc00c"
server_key_iv: "39aa9575892624135a3a9df1da459efa"
server_key_version: "encryptionkey@0"
I20260620 06:06:44.969554 11920 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781935603120653-11672-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781935603120653-11672-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781935603120653-11672-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:06:44.997901 11920 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:06:45.001050 11920 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:06:45.001250 11920 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:06:45.001833 11920 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 06:06:45.002624 11920 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 06:06:45.002669 11920 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 06:06:45.002718 11920 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 06:06:45.002774 11920 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 06:06:45.012423 11920 rpc_server.cc:307] RPC server started. Bound to: 127.11.102.2:38071
I20260620 06:06:45.012450 12048 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.102.2:38071 every 8 connection(s)
I20260620 06:06:45.013296 11920 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781935603120653-11672-0/minicluster-data/ts-1/data/info.pb
I20260620 06:06:45.017735 11672 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9HMp43/build/debug/bin/kudu as pid 11920
I20260620 06:06:45.017858 11672 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781935603120653-11672-0/minicluster-data/ts-1/wal/instance
I20260620 06:06:45.018108 11672 external_mini_cluster.cc:1428] Setting key 70f83a3c08e27b62d1e6d6ab5d55ea26
Jun 20 06:06:45 dist-test-slave-9hm5 krb5kdc[11679](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935604, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.102.2@KRBTEST.COM for kudu/127.11.102.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.11.102.3@KRBTEST.COM; defaulting to no policy
I20260620 06:06:45.026999 11763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:06:45.015024 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.102.2:44811 (local address 127.11.102.62:44285)
0620 06:06:45.015155 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0620 06:06:45.015164 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 06:06:45.016161 (+ 997us) server_negotiation.cc:407] Connection header received
0620 06:06:45.017143 (+ 982us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:06:45.017146 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:06:45.017199 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:06:45.017308 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:06:45.018609 (+ 1301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:06:45.019321 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:06:45.020484 (+ 1163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:06:45.020722 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:06:45.020835 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:06:45.023465 (+ 2630us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:06:45.023469 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:06:45.023471 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:06:45.023496 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:06:45.025028 (+ 1532us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:06:45.025525 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:06:45.025530 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:06:45.025532 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:06:45.025620 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:06:45.025949 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:06:45.025954 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:06:45.025956 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:06:45.026154 (+ 198us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:06:45.026261 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0620 06:06:45.026721 (+ 460us) server_negotiation.cc:299] Negotiation successful
0620 06:06:45.026838 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260620 06:06:45.027818 12051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:06:45.015356 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.102.62:44285 (local address 127.11.102.2:44811)
0620 06:06:45.016011 (+ 655us) negotiation.cc:107] Waiting for socket to connect
0620 06:06:45.016055 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0620 06:06:45.016944 (+ 889us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:06:45.017469 (+ 525us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:06:45.017476 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:06:45.017970 (+ 494us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:06:45.018472 (+ 502us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:06:45.018485 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:06:45.019438 (+ 953us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:06:45.019443 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:06:45.020363 (+ 920us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:06:45.020372 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:06:45.020867 (+ 495us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:06:45.020872 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:06:45.021031 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:06:45.021615 (+ 584us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:06:45.021638 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:06:45.023310 (+ 1672us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:06:45.025151 (+ 1841us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:06:45.025156 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:06:45.025169 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:06:45.025424 (+ 255us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:06:45.025711 (+ 287us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:06:45.025715 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:06:45.025717 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:06:45.025853 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:06:45.026257 (+ 404us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:06:45.026263 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:06:45.026494 (+ 231us) client_negotiation.cc:769] Sending connection context
0620 06:06:45.026684 (+ 190us) client_negotiation.cc:240] Negotiation successful
0620 06:06:45.026980 (+ 296us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":484,"thread_start_us":110,"threads_started":1}
I20260620 06:06:45.028968 12049 heartbeater.cc:347] Connected to a master server at 127.11.102.62:44285
I20260620 06:06:45.029176 12049 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.102.3@KRBTEST.COM" created.
I20260620 06:06:45.029670 12049 heartbeater.cc:511] Master 127.11.102.62:44285 requested a full tablet report, sending...
I20260620 06:06:45.030705 11724 ts_manager.cc:194] Registered new tserver with Master: 9cd67b8a38e34e1985d33c4bfbb497eb (127.11.102.2:38071)
I20260620 06:06:45.031486 11724 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.102.2@KRBTEST.COM'} at 127.11.102.2:44811
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.102.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781935603120653-11672-0/krb5kdc/kudu_127.11.102.3.keytab.
Entry for principal kudu/127.11.102.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781935603120653-11672-0/krb5kdc/kudu_127.11.102.3.keytab.
I20260620 06:06:45.045957 11672 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9HMp43/build/debug/bin/kudu
/tmp/dist-test-task9HMp43/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781935603120653-11672-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781935603120653-11672-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781935603120653-11672-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781935603120653-11672-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.102.3:0
--local_ip_for_outbound_sockets=127.11.102.3
--webserver_interface=127.11.102.3
--webserver_port=0
--tserver_master_addrs=127.11.102.62:44285
--keytab_file=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781935603120653-11672-0/krb5kdc/kudu_127.11.102.3.keytab
--principal=kudu/127.11.102.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.102.20:43587
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781935603120653-11672-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781935603120653-11672-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781935603120653-11672-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 06:06:45.156687 12054 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:06:45.156944 12054 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:06:45.156991 12054 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:06:45.161036 12054 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:06:45.161095 12054 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:06:45.161216 12054 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.102.3
I20260620 06:06:45.166400 12054 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.102.20:43587
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781935603120653-11672-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781935603120653-11672-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781935603120653-11672-0/krb5kdc/kudu_127.11.102.3.keytab
--principal=kudu/127.11.102.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.102.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781935603120653-11672-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.11.102.3
--webserver_port=0
--tserver_master_addrs=127.11.102.62:44285
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12054
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.102.3
--log_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781935603120653-11672-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 05:28:48 UTC on ccycloud.quasar-fixhms.root.comops.site
build id 4457020
I20260620 06:06:45.167455 12054 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:06:45.168567 12054 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 06:06:45.177896 12062 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:06:45.177944 12059 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:06:45.178045 12054 server_base.cc:1034] running on GCE node
W20260620 06:06:45.178256 12060 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:06:45.178623 12054 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:06:45.179250 12054 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:06:45.180433 12054 hybrid_clock.cc:648] HybridClock initialized: now 1781935605180397 us; error 45 us; skew 500 ppm
Jun 20 06:06:45 dist-test-slave-9hm5 krb5kdc[11679](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935605, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.102.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 06:06:45.183327 12054 init.cc:377] Logged in from keytab as kudu/127.11.102.3@KRBTEST.COM (short username kudu)
I20260620 06:06:45.184268 12054 webserver.cc:460] Webserver started at http://127.11.102.3:37563/ using document root <none> and password file <none>
I20260620 06:06:45.184823 12054 fs_manager.cc:359] Metadata directory not provided
I20260620 06:06:45.184890 12054 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:06:45.185130 12054 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 06:06:45.187168 12054 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781935603120653-11672-0/minicluster-data/ts-2/data/instance:
uuid: "312ac30c7df24cefbd046f5da86fe08d"
format_stamp: "Formatted at 2026-06-20 06:06:45 on dist-test-slave-9hm5"
server_key: "68108e35d8a2edd827b5870f9c1eb173"
server_key_iv: "70ca375204900492b46c5e1fe1f03b1f"
server_key_version: "encryptionkey@0"
I20260620 06:06:45.187829 12054 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781935603120653-11672-0/minicluster-data/ts-2/wal/instance:
uuid: "312ac30c7df24cefbd046f5da86fe08d"
format_stamp: "Formatted at 2026-06-20 06:06:45 on dist-test-slave-9hm5"
server_key: "68108e35d8a2edd827b5870f9c1eb173"
server_key_iv: "70ca375204900492b46c5e1fe1f03b1f"
server_key_version: "encryptionkey@0"
I20260620 06:06:45.191681 12054 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260620 06:06:45.194191 12069 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:06:45.195436 12054 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 06:06:45.195537 12054 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781935603120653-11672-0/minicluster-data/ts-2/data,/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781935603120653-11672-0/minicluster-data/ts-2/wal
uuid: "312ac30c7df24cefbd046f5da86fe08d"
format_stamp: "Formatted at 2026-06-20 06:06:45 on dist-test-slave-9hm5"
server_key: "68108e35d8a2edd827b5870f9c1eb173"
server_key_iv: "70ca375204900492b46c5e1fe1f03b1f"
server_key_version: "encryptionkey@0"
I20260620 06:06:45.195709 12054 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781935603120653-11672-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781935603120653-11672-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781935603120653-11672-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:06:45.238775 12054 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:06:45.242724 12054 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:06:45.242967 12054 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:06:45.243659 12054 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 06:06:45.244553 12054 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 06:06:45.244604 12054 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 06:06:45.244674 12054 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 06:06:45.244712 12054 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 06:06:45.254612 12054 rpc_server.cc:307] RPC server started. Bound to: 127.11.102.3:46629
I20260620 06:06:45.254670 12182 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.102.3:46629 every 8 connection(s)
I20260620 06:06:45.255538 12054 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781935603120653-11672-0/minicluster-data/ts-2/data/info.pb
I20260620 06:06:45.262255 11672 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9HMp43/build/debug/bin/kudu as pid 12054
I20260620 06:06:45.262410 11672 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781935603120653-11672-0/minicluster-data/ts-2/wal/instance
I20260620 06:06:45.262728 11672 external_mini_cluster.cc:1428] Setting key 423aa41ff288c7f20d9fad25b6349b59
Jun 20 06:06:45 dist-test-slave-9hm5 krb5kdc[11679](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935605, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.102.3@KRBTEST.COM for kudu/127.11.102.62@KRBTEST.COM
I20260620 06:06:45.269779 11763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:06:45.257433 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.102.3:39227 (local address 127.11.102.62:44285)
0620 06:06:45.257576 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0620 06:06:45.257580 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 06:06:45.258563 (+ 983us) server_negotiation.cc:407] Connection header received
0620 06:06:45.259481 (+ 918us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:06:45.259485 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:06:45.259553 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:06:45.259699 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:06:45.260843 (+ 1144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:06:45.261376 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:06:45.262537 (+ 1161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:06:45.262907 (+ 370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:06:45.263067 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:06:45.266007 (+ 2940us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:06:45.266027 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:06:45.266030 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:06:45.266054 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:06:45.267416 (+ 1362us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:06:45.268104 (+ 688us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:06:45.268109 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:06:45.268111 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:06:45.268204 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:06:45.268601 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:06:45.268605 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:06:45.268607 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:06:45.268806 (+ 199us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:06:45.268915 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0620 06:06:45.269453 (+ 538us) server_negotiation.cc:299] Negotiation successful
0620 06:06:45.269587 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260620 06:06:45.270598 12185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:06:45.257751 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.102.62:44285 (local address 127.11.102.3:39227)
0620 06:06:45.258404 (+ 653us) negotiation.cc:107] Waiting for socket to connect
0620 06:06:45.258445 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0620 06:06:45.259271 (+ 826us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:06:45.259875 (+ 604us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:06:45.259881 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:06:45.260235 (+ 354us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:06:45.260698 (+ 463us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:06:45.260711 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:06:45.261510 (+ 799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:06:45.261519 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:06:45.262377 (+ 858us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:06:45.262411 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:06:45.263089 (+ 678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:06:45.263092 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:06:45.263295 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:06:45.264027 (+ 732us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:06:45.264049 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:06:45.265851 (+ 1802us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:06:45.267568 (+ 1717us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:06:45.267575 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:06:45.267591 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:06:45.267989 (+ 398us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:06:45.268306 (+ 317us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:06:45.268312 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:06:45.268314 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:06:45.268468 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:06:45.268926 (+ 458us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:06:45.268934 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:06:45.269191 (+ 257us) client_negotiation.cc:769] Sending connection context
0620 06:06:45.269396 (+ 205us) client_negotiation.cc:240] Negotiation successful
0620 06:06:45.269746 (+ 350us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":476,"thread_start_us":110,"threads_started":1}
I20260620 06:06:45.271723 12183 heartbeater.cc:347] Connected to a master server at 127.11.102.62:44285
I20260620 06:06:45.271967 12183 heartbeater.cc:464] Registering TS with master...
I20260620 06:06:45.272455 12183 heartbeater.cc:511] Master 127.11.102.62:44285 requested a full tablet report, sending...
I20260620 06:06:45.273536 11724 ts_manager.cc:194] Registered new tserver with Master: 312ac30c7df24cefbd046f5da86fe08d (127.11.102.3:46629)
I20260620 06:06:45.274165 11724 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.102.3@KRBTEST.COM'} at 127.11.102.3:39227
I20260620 06:06:45.277237 11672 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 20 06:06:45 dist-test-slave-9hm5 krb5kdc[11679](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935605, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 20 06:06:45 dist-test-slave-9hm5 krb5kdc[11679](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935605, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.11.102.1@KRBTEST.COM
I20260620 06:06:45.302939 12190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:06:45.293242 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57116 (local address 127.11.102.1:44961)
0620 06:06:45.293504 (+ 262us) server_negotiation.cc:206] Beginning negotiation
0620 06:06:45.293522 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0620 06:06:45.293556 (+ 34us) server_negotiation.cc:407] Connection header received
0620 06:06:45.293685 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:06:45.293690 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:06:45.293921 (+ 231us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:06:45.294071 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:06:45.294706 (+ 635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:06:45.295514 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:06:45.296364 (+ 850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:06:45.296740 (+ 376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:06:45.296829 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:06:45.299293 (+ 2464us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:06:45.299301 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:06:45.299317 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:06:45.299342 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:06:45.301148 (+ 1806us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:06:45.301585 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:06:45.301592 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:06:45.301594 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:06:45.301677 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:06:45.301947 (+ 270us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:06:45.301952 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:06:45.301954 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:06:45.302203 (+ 249us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:06:45.302439 (+ 236us) server_negotiation.cc:1035] Waiting for connection context
0620 06:06:45.302588 (+ 149us) server_negotiation.cc:299] Negotiation successful
0620 06:06:45.302754 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":166,"thread_start_us":99,"threads_started":1}
W20260620 06:06:45.303975 11829 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57116
Jun 20 06:06:45 dist-test-slave-9hm5 krb5kdc[11679](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935605, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 20 06:06:45 dist-test-slave-9hm5 krb5kdc[11679](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935605, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.102.1@KRBTEST.COM
I20260620 06:06:45.331264 12190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:06:45.322521 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57130 (local address 127.11.102.1:44961)
0620 06:06:45.322687 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0620 06:06:45.322691 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 06:06:45.322826 (+ 135us) server_negotiation.cc:407] Connection header received
0620 06:06:45.322987 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:06:45.322997 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:06:45.323038 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:06:45.323146 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:06:45.323865 (+ 719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:06:45.324386 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:06:45.325218 (+ 832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:06:45.325455 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:06:45.325565 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:06:45.327970 (+ 2405us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:06:45.327973 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:06:45.327975 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:06:45.327997 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:06:45.329399 (+ 1402us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:06:45.329872 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:06:45.329875 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:06:45.329876 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:06:45.329947 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:06:45.330236 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:06:45.330238 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:06:45.330239 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:06:45.330445 (+ 206us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:06:45.330615 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0620 06:06:45.330937 (+ 322us) server_negotiation.cc:299] Negotiation successful
0620 06:06:45.331057 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260620 06:06:45.332360 11672 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9HMp43/build/debug/bin/kudu with pid 11786
I20260620 06:06:45.338627 11672 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9HMp43/build/debug/bin/kudu with pid 11920
I20260620 06:06:45.344123 11672 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9HMp43/build/debug/bin/kudu with pid 12054
I20260620 06:06:45.349694 11672 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9HMp43/build/debug/bin/kudu with pid 11692
2026-06-20T06:06:45Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (2224 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781935603120653-11672-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 06:06:45 dist-test-slave-9hm5 krb5kdc[12197](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 06:06:45 dist-test-slave-9hm5 krb5kdc[12197](info): set up 2 sockets
Jun 20 06:06:45 dist-test-slave-9hm5 krb5kdc[12197](info): commencing operation
krb5kdc: starting...
W20260620 06:06:46.399158 11672 mini_kdc.cc:121] Time spent starting KDC: real 1.027s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781935603120653-11672-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781935603120653-11672-0/krb5kdc/test-user.keytab.
Jun 20 06:06:46 dist-test-slave-9hm5 krb5kdc[12197](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935606, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T06:06:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T06:06:46Z Disabled control of system clock
WARNING: no policy specified for kudu/127.11.102.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.102.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.102.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781935603120653-11672-0/krb5kdc/kudu_127.11.102.62.keytab.
Entry for principal kudu/127.11.102.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781935603120653-11672-0/krb5kdc/kudu_127.11.102.62.keytab.
I20260620 06:06:46.515579 11672 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9HMp43/build/debug/bin/kudu
/tmp/dist-test-task9HMp43/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781935603120653-11672-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781935603120653-11672-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781935603120653-11672-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781935603120653-11672-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.102.62:39097
--webserver_interface=127.11.102.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781935603120653-11672-0/krb5kdc/kudu_127.11.102.62.keytab
--principal=kudu/127.11.102.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.102.20:42433
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.102.62:39097
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781935603120653-11672-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781935603120653-11672-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781935603120653-11672-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 06:06:46.620698 12210 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:06:46.620978 12210 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:06:46.621052 12210 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:06:46.624990 12210 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 06:06:46.625084 12210 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:06:46.625123 12210 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:06:46.625160 12210 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 06:06:46.625203 12210 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 06:06:46.630367 12210 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.102.20:42433
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781935603120653-11672-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781935603120653-11672-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.102.62:39097
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781935603120653-11672-0/krb5kdc/kudu_127.11.102.62.keytab
--principal=kudu/127.11.102.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.102.62:39097
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781935603120653-11672-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.102.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12210
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781935603120653-11672-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 05:28:48 UTC on ccycloud.quasar-fixhms.root.comops.site
build id 4457020
I20260620 06:06:46.631412 12210 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:06:46.632463 12210 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 06:06:46.640497 12218 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:06:46.640475 12216 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:06:46.640553 12210 server_base.cc:1034] running on GCE node
W20260620 06:06:46.640497 12215 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:06:46.641283 12210 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:06:46.642395 12210 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:06:46.643554 12210 hybrid_clock.cc:648] HybridClock initialized: now 1781935606643531 us; error 38 us; skew 500 ppm
Jun 20 06:06:46 dist-test-slave-9hm5 krb5kdc[12197](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935606, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.102.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 06:06:46.646348 12210 init.cc:377] Logged in from keytab as kudu/127.11.102.62@KRBTEST.COM (short username kudu)
I20260620 06:06:46.647204 12210 webserver.cc:460] Webserver started at http://127.11.102.62:42447/ using document root <none> and password file <none>
I20260620 06:06:46.647807 12210 fs_manager.cc:359] Metadata directory not provided
I20260620 06:06:46.647869 12210 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:06:46.648074 12210 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 06:06:46.650044 12210 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781935603120653-11672-0/minicluster-data/master-0/data/instance:
uuid: "71c554eca3b04c2f91531c2da50eb4f9"
format_stamp: "Formatted at 2026-06-20 06:06:46 on dist-test-slave-9hm5"
server_key: "8c24613bf099827c24d761d21e76c29b"
server_key_iv: "c6da7ee687e0a6dea6b0efd516a7ff92"
server_key_version: "encryptionkey@0"
I20260620 06:06:46.650622 12210 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781935603120653-11672-0/minicluster-data/master-0/wal/instance:
uuid: "71c554eca3b04c2f91531c2da50eb4f9"
format_stamp: "Formatted at 2026-06-20 06:06:46 on dist-test-slave-9hm5"
server_key: "8c24613bf099827c24d761d21e76c29b"
server_key_iv: "c6da7ee687e0a6dea6b0efd516a7ff92"
server_key_version: "encryptionkey@0"
I20260620 06:06:46.654299 12210 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260620 06:06:46.656648 12225 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:06:46.657838 12210 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 06:06:46.657979 12210 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781935603120653-11672-0/minicluster-data/master-0/data,/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781935603120653-11672-0/minicluster-data/master-0/wal
uuid: "71c554eca3b04c2f91531c2da50eb4f9"
format_stamp: "Formatted at 2026-06-20 06:06:46 on dist-test-slave-9hm5"
server_key: "8c24613bf099827c24d761d21e76c29b"
server_key_iv: "c6da7ee687e0a6dea6b0efd516a7ff92"
server_key_version: "encryptionkey@0"
I20260620 06:06:46.658090 12210 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781935603120653-11672-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781935603120653-11672-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781935603120653-11672-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:06:46.671525 12210 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:06:46.674290 12210 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:06:46.674482 12210 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:06:46.681517 12210 rpc_server.cc:307] RPC server started. Bound to: 127.11.102.62:39097
I20260620 06:06:46.681536 12277 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.102.62:39097 every 8 connection(s)
I20260620 06:06:46.682407 12210 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781935603120653-11672-0/minicluster-data/master-0/data/info.pb
I20260620 06:06:46.685068 12278 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 06:06:46.690290 12278 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 71c554eca3b04c2f91531c2da50eb4f9: Bootstrap starting.
I20260620 06:06:46.691216 11672 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9HMp43/build/debug/bin/kudu as pid 12210
I20260620 06:06:46.691325 11672 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781935603120653-11672-0/minicluster-data/master-0/wal/instance
I20260620 06:06:46.691596 11672 external_mini_cluster.cc:1428] Setting key a60e4b11dab3a8560efd4bf8345ce8b1
I20260620 06:06:46.693204 12278 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 71c554eca3b04c2f91531c2da50eb4f9: Neither blocks nor log segments found. Creating new log.
I20260620 06:06:46.693921 12278 log.cc:824] T 00000000000000000000000000000000 P 71c554eca3b04c2f91531c2da50eb4f9: Log is configured to *not* fsync() on all Append() calls
I20260620 06:06:46.696141 12278 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 71c554eca3b04c2f91531c2da50eb4f9: No bootstrap required, opened a new log
I20260620 06:06:46.700441 12278 raft_consensus.cc:348] T 00000000000000000000000000000000 P 71c554eca3b04c2f91531c2da50eb4f9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "71c554eca3b04c2f91531c2da50eb4f9" member_type: VOTER last_known_addr { host: "127.11.102.62" port: 39097 } }
I20260620 06:06:46.700652 12278 raft_consensus.cc:374] T 00000000000000000000000000000000 P 71c554eca3b04c2f91531c2da50eb4f9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 06:06:46.700695 12278 raft_consensus.cc:729] T 00000000000000000000000000000000 P 71c554eca3b04c2f91531c2da50eb4f9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 71c554eca3b04c2f91531c2da50eb4f9, State: Initialized, Role: FOLLOWER
Jun 20 06:06:46 dist-test-slave-9hm5 krb5kdc[12197](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935606, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.102.62@KRBTEST.COM
I20260620 06:06:46.701182 12278 consensus_queue.cc:260] T 00000000000000000000000000000000 P 71c554eca3b04c2f91531c2da50eb4f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "71c554eca3b04c2f91531c2da50eb4f9" member_type: VOTER last_known_addr { host: "127.11.102.62" port: 39097 } }
I20260620 06:06:46.701352 12278 raft_consensus.cc:388] T 00000000000000000000000000000000 P 71c554eca3b04c2f91531c2da50eb4f9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 06:06:46.701436 12278 raft_consensus.cc:482] T 00000000000000000000000000000000 P 71c554eca3b04c2f91531c2da50eb4f9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 06:06:46.701553 12278 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 71c554eca3b04c2f91531c2da50eb4f9 [term 0 FOLLOWER]: Advancing to term 1
I20260620 06:06:46.702437 12278 raft_consensus.cc:504] T 00000000000000000000000000000000 P 71c554eca3b04c2f91531c2da50eb4f9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "71c554eca3b04c2f91531c2da50eb4f9" member_type: VOTER last_known_addr { host: "127.11.102.62" port: 39097 } }
I20260620 06:06:46.702790 12278 leader_election.cc:302] T 00000000000000000000000000000000 P 71c554eca3b04c2f91531c2da50eb4f9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 71c554eca3b04c2f91531c2da50eb4f9; no voters:
I20260620 06:06:46.703179 12278 leader_election.cc:288] T 00000000000000000000000000000000 P 71c554eca3b04c2f91531c2da50eb4f9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 06:06:46.703260 12283 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 71c554eca3b04c2f91531c2da50eb4f9 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 06:06:46.703472 12283 raft_consensus.cc:686] T 00000000000000000000000000000000 P 71c554eca3b04c2f91531c2da50eb4f9 [term 1 LEADER]: Becoming Leader. State: Replica: 71c554eca3b04c2f91531c2da50eb4f9, State: Running, Role: LEADER
I20260620 06:06:46.703851 12283 consensus_queue.cc:237] T 00000000000000000000000000000000 P 71c554eca3b04c2f91531c2da50eb4f9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "71c554eca3b04c2f91531c2da50eb4f9" member_type: VOTER last_known_addr { host: "127.11.102.62" port: 39097 } }
I20260620 06:06:46.704519 12278 sys_catalog.cc:564] T 00000000000000000000000000000000 P 71c554eca3b04c2f91531c2da50eb4f9 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 06:06:46.705909 12285 sys_catalog.cc:455] T 00000000000000000000000000000000 P 71c554eca3b04c2f91531c2da50eb4f9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 71c554eca3b04c2f91531c2da50eb4f9. Latest consensus state: current_term: 1 leader_uuid: "71c554eca3b04c2f91531c2da50eb4f9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "71c554eca3b04c2f91531c2da50eb4f9" member_type: VOTER last_known_addr { host: "127.11.102.62" port: 39097 } } }
I20260620 06:06:46.706116 12285 sys_catalog.cc:458] T 00000000000000000000000000000000 P 71c554eca3b04c2f91531c2da50eb4f9 [sys.catalog]: This master's current role is: LEADER
I20260620 06:06:46.705952 12284 sys_catalog.cc:455] T 00000000000000000000000000000000 P 71c554eca3b04c2f91531c2da50eb4f9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "71c554eca3b04c2f91531c2da50eb4f9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "71c554eca3b04c2f91531c2da50eb4f9" member_type: VOTER last_known_addr { host: "127.11.102.62" port: 39097 } } }
I20260620 06:06:46.706297 12284 sys_catalog.cc:458] T 00000000000000000000000000000000 P 71c554eca3b04c2f91531c2da50eb4f9 [sys.catalog]: This master's current role is: LEADER
I20260620 06:06:46.706976 12294 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 06:06:46.707799 12281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:06:46.692783 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47430 (local address 127.11.102.62:39097)
0620 06:06:46.693206 (+ 423us) server_negotiation.cc:206] Beginning negotiation
0620 06:06:46.693223 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0620 06:06:46.693282 (+ 59us) server_negotiation.cc:407] Connection header received
0620 06:06:46.693965 (+ 683us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:06:46.694009 (+ 44us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:06:46.694449 (+ 440us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:06:46.694818 (+ 369us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:06:46.695722 (+ 904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:06:46.696782 (+ 1060us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:06:46.697958 (+ 1176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:06:46.698454 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:06:46.698578 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:06:46.701706 (+ 3128us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:06:46.701711 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:06:46.701723 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:06:46.701749 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:06:46.703783 (+ 2034us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:06:46.704269 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:06:46.704274 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:06:46.704278 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:06:46.704383 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:06:46.704728 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:06:46.704731 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:06:46.704733 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:06:46.705251 (+ 518us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:06:46.706213 (+ 962us) server_negotiation.cc:1035] Waiting for connection context
0620 06:06:46.706418 (+ 205us) server_negotiation.cc:299] Negotiation successful
0620 06:06:46.706751 (+ 333us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":261,"thread_start_us":137,"threads_started":1}
I20260620 06:06:46.710112 12294 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 06:06:46.715251 12294 catalog_manager.cc:1347] Generated new cluster ID: abdd738ffb6f4971b7e9595702a7942b
I20260620 06:06:46.715314 12294 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 06:06:46.733100 12294 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 06:06:46.734272 12294 catalog_manager.cc:1504] Loading token signing keys...
I20260620 06:06:46.745397 12294 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 71c554eca3b04c2f91531c2da50eb4f9: Generated new TSK 0
I20260620 06:06:46.746217 12294 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.11.102.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.102.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.102.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781935603120653-11672-0/krb5kdc/kudu_127.11.102.1.keytab.
Entry for principal kudu/127.11.102.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781935603120653-11672-0/krb5kdc/kudu_127.11.102.1.keytab.
I20260620 06:06:46.774564 11672 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9HMp43/build/debug/bin/kudu
/tmp/dist-test-task9HMp43/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781935603120653-11672-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781935603120653-11672-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781935603120653-11672-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781935603120653-11672-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.102.1:0
--local_ip_for_outbound_sockets=127.11.102.1
--webserver_interface=127.11.102.1
--webserver_port=0
--tserver_master_addrs=127.11.102.62:39097
--keytab_file=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781935603120653-11672-0/krb5kdc/kudu_127.11.102.1.keytab
--principal=kudu/127.11.102.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.102.20:42433
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781935603120653-11672-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781935603120653-11672-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781935603120653-11672-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 06:06:46.885442 12304 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:06:46.885705 12304 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:06:46.885751 12304 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:06:46.889703 12304 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:06:46.889757 12304 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:06:46.889818 12304 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.102.1
I20260620 06:06:46.894801 12304 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.102.20:42433
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781935603120653-11672-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781935603120653-11672-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781935603120653-11672-0/krb5kdc/kudu_127.11.102.1.keytab
--principal=kudu/127.11.102.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.102.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781935603120653-11672-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.11.102.1
--webserver_port=0
--tserver_master_addrs=127.11.102.62:39097
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12304
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.102.1
--log_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781935603120653-11672-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 05:28:48 UTC on ccycloud.quasar-fixhms.root.comops.site
build id 4457020
I20260620 06:06:46.895737 12304 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:06:46.896670 12304 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 06:06:46.906358 12310 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:06:46.906332 12309 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:06:46.906328 12312 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:06:46.906520 12304 server_base.cc:1034] running on GCE node
I20260620 06:06:46.907001 12304 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:06:46.907675 12304 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:06:46.908864 12304 hybrid_clock.cc:648] HybridClock initialized: now 1781935606908834 us; error 44 us; skew 500 ppm
Jun 20 06:06:46 dist-test-slave-9hm5 krb5kdc[12197](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935606, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.102.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 06:06:46.911806 12304 init.cc:377] Logged in from keytab as kudu/127.11.102.1@KRBTEST.COM (short username kudu)
I20260620 06:06:46.912592 12304 webserver.cc:460] Webserver started at http://127.11.102.1:37269/ using document root <none> and password file <none>
I20260620 06:06:46.913096 12304 fs_manager.cc:359] Metadata directory not provided
I20260620 06:06:46.913159 12304 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:06:46.913322 12304 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 06:06:46.915378 12304 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781935603120653-11672-0/minicluster-data/ts-0/data/instance:
uuid: "3f1f48c18638443383a6cac67477eb85"
format_stamp: "Formatted at 2026-06-20 06:06:46 on dist-test-slave-9hm5"
server_key: "d0693a1019dc327289d775eca83e8c9a"
server_key_iv: "0ed706f1153a4cf1481ae06fe068a6db"
server_key_version: "encryptionkey@0"
I20260620 06:06:46.915958 12304 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781935603120653-11672-0/minicluster-data/ts-0/wal/instance:
uuid: "3f1f48c18638443383a6cac67477eb85"
format_stamp: "Formatted at 2026-06-20 06:06:46 on dist-test-slave-9hm5"
server_key: "d0693a1019dc327289d775eca83e8c9a"
server_key_iv: "0ed706f1153a4cf1481ae06fe068a6db"
server_key_version: "encryptionkey@0"
I20260620 06:06:46.919584 12304 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260620 06:06:46.922042 12319 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:06:46.923349 12304 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 06:06:46.923492 12304 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781935603120653-11672-0/minicluster-data/ts-0/data,/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781935603120653-11672-0/minicluster-data/ts-0/wal
uuid: "3f1f48c18638443383a6cac67477eb85"
format_stamp: "Formatted at 2026-06-20 06:06:46 on dist-test-slave-9hm5"
server_key: "d0693a1019dc327289d775eca83e8c9a"
server_key_iv: "0ed706f1153a4cf1481ae06fe068a6db"
server_key_version: "encryptionkey@0"
I20260620 06:06:46.923584 12304 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781935603120653-11672-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781935603120653-11672-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781935603120653-11672-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:06:46.951211 12304 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:06:46.954699 12304 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:06:46.954924 12304 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:06:46.955542 12304 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 06:06:46.956439 12304 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 06:06:46.956488 12304 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 06:06:46.956560 12304 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 06:06:46.956597 12304 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 06:06:46.966334 12304 rpc_server.cc:307] RPC server started. Bound to: 127.11.102.1:40869
I20260620 06:06:46.966367 12432 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.102.1:40869 every 8 connection(s)
I20260620 06:06:46.967200 12304 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781935603120653-11672-0/minicluster-data/ts-0/data/info.pb
I20260620 06:06:46.970582 11672 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9HMp43/build/debug/bin/kudu as pid 12304
I20260620 06:06:46.970664 11672 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781935603120653-11672-0/minicluster-data/ts-0/wal/instance
I20260620 06:06:46.970912 11672 external_mini_cluster.cc:1428] Setting key fa43103a33f61858a3fd5fc68214a6b0
Jun 20 06:06:46 dist-test-slave-9hm5 krb5kdc[12197](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935606, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.102.1@KRBTEST.COM for kudu/127.11.102.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.11.102.2@KRBTEST.COM; defaulting to no policy
I20260620 06:06:46.981511 12281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:06:46.968913 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.102.1:38699 (local address 127.11.102.62:39097)
0620 06:06:46.969041 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0620 06:06:46.969045 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 06:06:46.970065 (+ 1020us) server_negotiation.cc:407] Connection header received
0620 06:06:46.971002 (+ 937us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:06:46.971006 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:06:46.971059 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:06:46.971148 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:06:46.972613 (+ 1465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:06:46.973487 (+ 874us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:06:46.974670 (+ 1183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:06:46.974912 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:06:46.975023 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:06:46.977890 (+ 2867us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:06:46.977893 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:06:46.977896 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:06:46.977917 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:06:46.979350 (+ 1433us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:06:46.979950 (+ 600us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:06:46.979952 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:06:46.979954 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:06:46.980025 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:06:46.980410 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:06:46.980412 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:06:46.980413 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:06:46.980569 (+ 156us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:06:46.980640 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0620 06:06:46.981284 (+ 644us) server_negotiation.cc:299] Negotiation successful
0620 06:06:46.981386 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260620 06:06:46.982235 12435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:06:46.969291 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.102.62:39097 (local address 127.11.102.1:38699)
0620 06:06:46.969928 (+ 637us) negotiation.cc:107] Waiting for socket to connect
0620 06:06:46.969968 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0620 06:06:46.970806 (+ 838us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:06:46.971359 (+ 553us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:06:46.971368 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:06:46.971875 (+ 507us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:06:46.972480 (+ 605us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:06:46.972494 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:06:46.973612 (+ 1118us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:06:46.973614 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:06:46.974550 (+ 936us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:06:46.974558 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:06:46.975055 (+ 497us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:06:46.975059 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:06:46.975308 (+ 249us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:06:46.975939 (+ 631us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:06:46.975963 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:06:46.977716 (+ 1753us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:06:46.979468 (+ 1752us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:06:46.979476 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:06:46.979493 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:06:46.979849 (+ 356us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:06:46.980120 (+ 271us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:06:46.980123 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:06:46.980125 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:06:46.980312 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:06:46.980674 (+ 362us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:06:46.980683 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:06:46.980995 (+ 312us) client_negotiation.cc:769] Sending connection context
0620 06:06:46.981183 (+ 188us) client_negotiation.cc:240] Negotiation successful
0620 06:06:46.981413 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":458,"thread_start_us":112,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.102.2@KRBTEST.COM" created.
I20260620 06:06:46.983677 12433 heartbeater.cc:347] Connected to a master server at 127.11.102.62:39097
I20260620 06:06:46.983923 12433 heartbeater.cc:464] Registering TS with master...
I20260620 06:06:46.984391 12433 heartbeater.cc:511] Master 127.11.102.62:39097 requested a full tablet report, sending...
I20260620 06:06:46.985859 12242 ts_manager.cc:194] Registered new tserver with Master: 3f1f48c18638443383a6cac67477eb85 (127.11.102.1:40869)
I20260620 06:06:46.987053 12242 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.102.1@KRBTEST.COM'} at 127.11.102.1:38699
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.102.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781935603120653-11672-0/krb5kdc/kudu_127.11.102.2.keytab.
Entry for principal kudu/127.11.102.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781935603120653-11672-0/krb5kdc/kudu_127.11.102.2.keytab.
I20260620 06:06:46.997879 11672 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9HMp43/build/debug/bin/kudu
/tmp/dist-test-task9HMp43/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781935603120653-11672-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781935603120653-11672-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781935603120653-11672-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781935603120653-11672-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.102.2:0
--local_ip_for_outbound_sockets=127.11.102.2
--webserver_interface=127.11.102.2
--webserver_port=0
--tserver_master_addrs=127.11.102.62:39097
--keytab_file=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781935603120653-11672-0/krb5kdc/kudu_127.11.102.2.keytab
--principal=kudu/127.11.102.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.102.20:42433
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781935603120653-11672-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781935603120653-11672-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781935603120653-11672-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 06:06:47.103500 12438 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:06:47.103824 12438 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:06:47.103911 12438 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:06:47.108125 12438 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:06:47.108207 12438 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:06:47.108296 12438 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.102.2
I20260620 06:06:47.113391 12438 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.102.20:42433
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781935603120653-11672-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781935603120653-11672-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781935603120653-11672-0/krb5kdc/kudu_127.11.102.2.keytab
--principal=kudu/127.11.102.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.102.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781935603120653-11672-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.11.102.2
--webserver_port=0
--tserver_master_addrs=127.11.102.62:39097
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12438
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.102.2
--log_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781935603120653-11672-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 05:28:48 UTC on ccycloud.quasar-fixhms.root.comops.site
build id 4457020
I20260620 06:06:47.114382 12438 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:06:47.115446 12438 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 06:06:47.124627 12444 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:06:47.124662 12443 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:06:47.124802 12438 server_base.cc:1034] running on GCE node
W20260620 06:06:47.124648 12446 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:06:47.125244 12438 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:06:47.125823 12438 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:06:47.126981 12438 hybrid_clock.cc:648] HybridClock initialized: now 1781935607126952 us; error 43 us; skew 500 ppm
Jun 20 06:06:47 dist-test-slave-9hm5 krb5kdc[12197](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935607, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.102.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 06:06:47.129920 12438 init.cc:377] Logged in from keytab as kudu/127.11.102.2@KRBTEST.COM (short username kudu)
I20260620 06:06:47.130800 12438 webserver.cc:460] Webserver started at http://127.11.102.2:33633/ using document root <none> and password file <none>
I20260620 06:06:47.131377 12438 fs_manager.cc:359] Metadata directory not provided
I20260620 06:06:47.131470 12438 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:06:47.131753 12438 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 06:06:47.133852 12438 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781935603120653-11672-0/minicluster-data/ts-1/data/instance:
uuid: "5d62c8e649da4ec6ab8e6ee19b2210d4"
format_stamp: "Formatted at 2026-06-20 06:06:47 on dist-test-slave-9hm5"
server_key: "8b8804c3498b2a6ddc3b1ea69524a915"
server_key_iv: "8bc7a1715b419f832d54b0269e9b1cb2"
server_key_version: "encryptionkey@0"
I20260620 06:06:47.134462 12438 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781935603120653-11672-0/minicluster-data/ts-1/wal/instance:
uuid: "5d62c8e649da4ec6ab8e6ee19b2210d4"
format_stamp: "Formatted at 2026-06-20 06:06:47 on dist-test-slave-9hm5"
server_key: "8b8804c3498b2a6ddc3b1ea69524a915"
server_key_iv: "8bc7a1715b419f832d54b0269e9b1cb2"
server_key_version: "encryptionkey@0"
I20260620 06:06:47.138193 12438 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260620 06:06:47.140715 12453 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:06:47.141968 12438 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 06:06:47.142115 12438 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781935603120653-11672-0/minicluster-data/ts-1/data,/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781935603120653-11672-0/minicluster-data/ts-1/wal
uuid: "5d62c8e649da4ec6ab8e6ee19b2210d4"
format_stamp: "Formatted at 2026-06-20 06:06:47 on dist-test-slave-9hm5"
server_key: "8b8804c3498b2a6ddc3b1ea69524a915"
server_key_iv: "8bc7a1715b419f832d54b0269e9b1cb2"
server_key_version: "encryptionkey@0"
I20260620 06:06:47.142246 12438 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781935603120653-11672-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781935603120653-11672-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781935603120653-11672-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:06:47.163722 12438 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:06:47.167114 12438 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:06:47.167326 12438 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:06:47.167990 12438 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 06:06:47.168836 12438 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 06:06:47.168907 12438 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 06:06:47.168977 12438 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 06:06:47.169029 12438 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 06:06:47.178328 12438 rpc_server.cc:307] RPC server started. Bound to: 127.11.102.2:42273
I20260620 06:06:47.178356 12566 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.102.2:42273 every 8 connection(s)
I20260620 06:06:47.179188 12438 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781935603120653-11672-0/minicluster-data/ts-1/data/info.pb
I20260620 06:06:47.183360 11672 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9HMp43/build/debug/bin/kudu as pid 12438
I20260620 06:06:47.183457 11672 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781935603120653-11672-0/minicluster-data/ts-1/wal/instance
I20260620 06:06:47.183763 11672 external_mini_cluster.cc:1428] Setting key a1a22ee963a10047f611348cbf0e833f
Jun 20 06:06:47 dist-test-slave-9hm5 krb5kdc[12197](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935607, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.102.2@KRBTEST.COM for kudu/127.11.102.62@KRBTEST.COM
I20260620 06:06:47.193425 12281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:06:47.180906 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.102.2:41639 (local address 127.11.102.62:39097)
0620 06:06:47.181042 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0620 06:06:47.181046 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 06:06:47.181835 (+ 789us) server_negotiation.cc:407] Connection header received
0620 06:06:47.182733 (+ 898us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:06:47.182736 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:06:47.182781 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:06:47.182858 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:06:47.184238 (+ 1380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:06:47.184974 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:06:47.186652 (+ 1678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:06:47.186920 (+ 268us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:06:47.187038 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:06:47.189511 (+ 2473us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:06:47.189518 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:06:47.189525 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:06:47.189559 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:06:47.191290 (+ 1731us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:06:47.191840 (+ 550us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:06:47.191845 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:06:47.191847 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:06:47.191943 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:06:47.192302 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:06:47.192305 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:06:47.192307 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:06:47.192536 (+ 229us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:06:47.192645 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0620 06:06:47.193118 (+ 473us) server_negotiation.cc:299] Negotiation successful
0620 06:06:47.193263 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
WARNING: no policy specified for kudu/127.11.102.3@KRBTEST.COM; defaulting to no policy
I20260620 06:06:47.194110 12569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:06:47.181230 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.102.62:39097 (local address 127.11.102.2:41639)
0620 06:06:47.181660 (+ 430us) negotiation.cc:107] Waiting for socket to connect
0620 06:06:47.181708 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0620 06:06:47.182521 (+ 813us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:06:47.183052 (+ 531us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:06:47.183059 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:06:47.183470 (+ 411us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:06:47.184088 (+ 618us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:06:47.184102 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:06:47.185109 (+ 1007us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:06:47.185113 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:06:47.186521 (+ 1408us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:06:47.186536 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:06:47.187071 (+ 535us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:06:47.187075 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:06:47.187247 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:06:47.187873 (+ 626us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:06:47.187889 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:06:47.189331 (+ 1442us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:06:47.191409 (+ 2078us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:06:47.191413 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:06:47.191423 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:06:47.191747 (+ 324us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:06:47.192063 (+ 316us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:06:47.192065 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:06:47.192066 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:06:47.192187 (+ 121us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:06:47.192637 (+ 450us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:06:47.192641 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:06:47.192859 (+ 218us) client_negotiation.cc:769] Sending connection context
0620 06:06:47.193001 (+ 142us) client_negotiation.cc:240] Negotiation successful
0620 06:06:47.193236 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":259,"thread_start_us":116,"threads_started":1}
I20260620 06:06:47.195317 12567 heartbeater.cc:347] Connected to a master server at 127.11.102.62:39097
I20260620 06:06:47.195528 12567 heartbeater.cc:464] Registering TS with master...
I20260620 06:06:47.196008 12567 heartbeater.cc:511] Master 127.11.102.62:39097 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.102.3@KRBTEST.COM" created.
I20260620 06:06:47.197088 12242 ts_manager.cc:194] Registered new tserver with Master: 5d62c8e649da4ec6ab8e6ee19b2210d4 (127.11.102.2:42273)
I20260620 06:06:47.197883 12242 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.102.2@KRBTEST.COM'} at 127.11.102.2:41639
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.102.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781935603120653-11672-0/krb5kdc/kudu_127.11.102.3.keytab.
Entry for principal kudu/127.11.102.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781935603120653-11672-0/krb5kdc/kudu_127.11.102.3.keytab.
I20260620 06:06:47.212028 11672 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9HMp43/build/debug/bin/kudu
/tmp/dist-test-task9HMp43/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781935603120653-11672-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781935603120653-11672-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781935603120653-11672-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781935603120653-11672-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.102.3:0
--local_ip_for_outbound_sockets=127.11.102.3
--webserver_interface=127.11.102.3
--webserver_port=0
--tserver_master_addrs=127.11.102.62:39097
--keytab_file=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781935603120653-11672-0/krb5kdc/kudu_127.11.102.3.keytab
--principal=kudu/127.11.102.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.102.20:42433
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781935603120653-11672-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781935603120653-11672-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781935603120653-11672-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 06:06:47.320987 12572 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:06:47.321256 12572 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:06:47.321339 12572 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:06:47.325362 12572 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:06:47.325440 12572 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:06:47.325527 12572 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.102.3
I20260620 06:06:47.330835 12572 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.102.20:42433
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781935603120653-11672-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781935603120653-11672-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781935603120653-11672-0/krb5kdc/kudu_127.11.102.3.keytab
--principal=kudu/127.11.102.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.102.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781935603120653-11672-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.11.102.3
--webserver_port=0
--tserver_master_addrs=127.11.102.62:39097
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12572
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.102.3
--log_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781935603120653-11672-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 05:28:48 UTC on ccycloud.quasar-fixhms.root.comops.site
build id 4457020
I20260620 06:06:47.331881 12572 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:06:47.332907 12572 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 06:06:47.342324 12580 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:06:47.342345 12578 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:06:47.342500 12572 server_base.cc:1034] running on GCE node
W20260620 06:06:47.342339 12577 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:06:47.342947 12572 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:06:47.343501 12572 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:06:47.344655 12572 hybrid_clock.cc:648] HybridClock initialized: now 1781935607344642 us; error 25 us; skew 500 ppm
Jun 20 06:06:47 dist-test-slave-9hm5 krb5kdc[12197](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935607, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.102.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 06:06:47.347568 12572 init.cc:377] Logged in from keytab as kudu/127.11.102.3@KRBTEST.COM (short username kudu)
I20260620 06:06:47.348519 12572 webserver.cc:460] Webserver started at http://127.11.102.3:46391/ using document root <none> and password file <none>
I20260620 06:06:47.349095 12572 fs_manager.cc:359] Metadata directory not provided
I20260620 06:06:47.349188 12572 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:06:47.349426 12572 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 06:06:47.351464 12572 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781935603120653-11672-0/minicluster-data/ts-2/data/instance:
uuid: "1fc1ea4cbd454945817965da6a7e9270"
format_stamp: "Formatted at 2026-06-20 06:06:47 on dist-test-slave-9hm5"
server_key: "24be1a4552370089a086ceed490c6b35"
server_key_iv: "a6b78a785c707feb2935604364eac35b"
server_key_version: "encryptionkey@0"
I20260620 06:06:47.352106 12572 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781935603120653-11672-0/minicluster-data/ts-2/wal/instance:
uuid: "1fc1ea4cbd454945817965da6a7e9270"
format_stamp: "Formatted at 2026-06-20 06:06:47 on dist-test-slave-9hm5"
server_key: "24be1a4552370089a086ceed490c6b35"
server_key_iv: "a6b78a785c707feb2935604364eac35b"
server_key_version: "encryptionkey@0"
I20260620 06:06:47.355785 12572 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 06:06:47.358223 12587 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:06:47.359432 12572 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 06:06:47.359562 12572 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781935603120653-11672-0/minicluster-data/ts-2/data,/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781935603120653-11672-0/minicluster-data/ts-2/wal
uuid: "1fc1ea4cbd454945817965da6a7e9270"
format_stamp: "Formatted at 2026-06-20 06:06:47 on dist-test-slave-9hm5"
server_key: "24be1a4552370089a086ceed490c6b35"
server_key_iv: "a6b78a785c707feb2935604364eac35b"
server_key_version: "encryptionkey@0"
I20260620 06:06:47.359727 12572 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781935603120653-11672-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781935603120653-11672-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781935603120653-11672-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:06:47.374046 12572 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:06:47.377404 12572 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:06:47.377600 12572 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:06:47.378197 12572 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 06:06:47.379035 12572 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 06:06:47.379107 12572 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 06:06:47.379179 12572 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 06:06:47.379223 12572 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 06:06:47.388463 12572 rpc_server.cc:307] RPC server started. Bound to: 127.11.102.3:45145
I20260620 06:06:47.388484 12700 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.102.3:45145 every 8 connection(s)
I20260620 06:06:47.389314 12572 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781935603120653-11672-0/minicluster-data/ts-2/data/info.pb
I20260620 06:06:47.397738 11672 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9HMp43/build/debug/bin/kudu as pid 12572
I20260620 06:06:47.397871 11672 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781935603120653-11672-0/minicluster-data/ts-2/wal/instance
I20260620 06:06:47.398139 11672 external_mini_cluster.cc:1428] Setting key 0e94306f781d2aa38aace4c76326411f
Jun 20 06:06:47 dist-test-slave-9hm5 krb5kdc[12197](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935607, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.102.3@KRBTEST.COM for kudu/127.11.102.62@KRBTEST.COM
I20260620 06:06:47.402881 12281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:06:47.391071 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.102.3:34381 (local address 127.11.102.62:39097)
0620 06:06:47.391233 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0620 06:06:47.391237 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 06:06:47.392113 (+ 876us) server_negotiation.cc:407] Connection header received
0620 06:06:47.393372 (+ 1259us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:06:47.393375 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:06:47.393420 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:06:47.393496 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:06:47.394652 (+ 1156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:06:47.395165 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:06:47.396391 (+ 1226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:06:47.396594 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:06:47.396666 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:06:47.399360 (+ 2694us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:06:47.399365 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:06:47.399367 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:06:47.399391 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:06:47.400895 (+ 1504us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:06:47.401411 (+ 516us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:06:47.401416 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:06:47.401418 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:06:47.401496 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:06:47.401831 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:06:47.401835 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:06:47.401836 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:06:47.402010 (+ 174us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:06:47.402113 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0620 06:06:47.402595 (+ 482us) server_negotiation.cc:299] Negotiation successful
0620 06:06:47.402718 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260620 06:06:47.403791 12703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:06:47.391461 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.102.62:39097 (local address 127.11.102.3:34381)
0620 06:06:47.391962 (+ 501us) negotiation.cc:107] Waiting for socket to connect
0620 06:06:47.392004 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0620 06:06:47.393181 (+ 1177us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:06:47.393677 (+ 496us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:06:47.393684 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:06:47.394051 (+ 367us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:06:47.394524 (+ 473us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:06:47.394537 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:06:47.395285 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:06:47.395288 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:06:47.396277 (+ 989us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:06:47.396288 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:06:47.396767 (+ 479us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:06:47.396770 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:06:47.396918 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:06:47.397459 (+ 541us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:06:47.397474 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:06:47.399191 (+ 1717us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:06:47.401011 (+ 1820us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:06:47.401017 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:06:47.401031 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:06:47.401306 (+ 275us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:06:47.401589 (+ 283us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:06:47.401593 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:06:47.401595 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:06:47.401733 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:06:47.402121 (+ 388us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:06:47.402130 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:06:47.402372 (+ 242us) client_negotiation.cc:769] Sending connection context
0620 06:06:47.402558 (+ 186us) client_negotiation.cc:240] Negotiation successful
0620 06:06:47.402861 (+ 303us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":317,"thread_start_us":163,"threads_started":1}
I20260620 06:06:47.404990 12701 heartbeater.cc:347] Connected to a master server at 127.11.102.62:39097
I20260620 06:06:47.405356 12701 heartbeater.cc:464] Registering TS with master...
I20260620 06:06:47.405862 12701 heartbeater.cc:511] Master 127.11.102.62:39097 requested a full tablet report, sending...
I20260620 06:06:47.406828 12242 ts_manager.cc:194] Registered new tserver with Master: 1fc1ea4cbd454945817965da6a7e9270 (127.11.102.3:45145)
I20260620 06:06:47.407336 12242 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.102.3@KRBTEST.COM'} at 127.11.102.3:34381
I20260620 06:06:47.412317 11672 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 06:06:47.422667 12281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:06:47.415753 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47444 (local address 127.11.102.62:39097)
0620 06:06:47.415890 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0620 06:06:47.415893 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 06:06:47.416005 (+ 112us) server_negotiation.cc:407] Connection header received
0620 06:06:47.416099 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:06:47.416101 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:06:47.416139 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:06:47.416209 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:06:47.416820 (+ 611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:06:47.417340 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:06:47.418141 (+ 801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:06:47.418355 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:06:47.418427 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:06:47.419849 (+ 1422us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:06:47.419851 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:06:47.419854 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:06:47.419874 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:06:47.421073 (+ 1199us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:06:47.421558 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:06:47.421561 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:06:47.421563 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:06:47.421654 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:06:47.421921 (+ 267us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:06:47.421924 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:06:47.421925 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:06:47.422116 (+ 191us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:06:47.422218 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0620 06:06:47.422467 (+ 249us) server_negotiation.cc:299] Negotiation successful
0620 06:06:47.422555 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260620 06:06:47.427399 12242 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47444:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 06:06:47.430485 12242 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 06:06:47.444490 12714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:06:47.438648 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43058 (local address 127.11.102.2:42273)
0620 06:06:47.439148 (+ 500us) server_negotiation.cc:206] Beginning negotiation
0620 06:06:47.439161 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 06:06:47.439185 (+ 24us) server_negotiation.cc:407] Connection header received
0620 06:06:47.439394 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:06:47.439401 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:06:47.439681 (+ 280us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:06:47.439844 (+ 163us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 06:06:47.440148 (+ 304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:06:47.441343 (+ 1195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:06:47.443024 (+ 1681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:06:47.443761 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:06:47.443872 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:06:47.444016 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0620 06:06:47.444226 (+ 210us) server_negotiation.cc:299] Negotiation successful
0620 06:06:47.444325 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":406,"thread_start_us":179,"threads_started":1}
I20260620 06:06:47.444510 12712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:06:47.438518 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.102.2:42273 (local address 127.0.0.1:43058)
0620 06:06:47.438984 (+ 466us) negotiation.cc:107] Waiting for socket to connect
0620 06:06:47.439041 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0620 06:06:47.439175 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:06:47.439827 (+ 652us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:06:47.439833 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:06:47.439855 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 06:06:47.440005 (+ 150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:06:47.440014 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:06:47.441492 (+ 1478us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:06:47.441495 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:06:47.442873 (+ 1378us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:06:47.442881 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:06:47.443887 (+ 1006us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:06:47.443891 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:06:47.444015 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:06:47.444052 (+ 37us) client_negotiation.cc:769] Sending connection context
0620 06:06:47.444183 (+ 131us) client_negotiation.cc:240] Negotiation successful
0620 06:06:47.444318 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":291,"mutex_wait_us":41,"thread_start_us":76,"threads_started":1}
I20260620 06:06:47.444847 12716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:06:47.439335 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34482 (local address 127.11.102.3:45145)
0620 06:06:47.439797 (+ 462us) server_negotiation.cc:206] Beginning negotiation
0620 06:06:47.439809 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 06:06:47.439830 (+ 21us) server_negotiation.cc:407] Connection header received
0620 06:06:47.439905 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:06:47.439909 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:06:47.440338 (+ 429us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:06:47.440505 (+ 167us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 06:06:47.440777 (+ 272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:06:47.441638 (+ 861us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:06:47.443010 (+ 1372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:06:47.443903 (+ 893us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:06:47.444004 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:06:47.444146 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0620 06:06:47.444578 (+ 432us) server_negotiation.cc:299] Negotiation successful
0620 06:06:47.444680 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":346,"thread_start_us":120,"threads_started":1}
I20260620 06:06:47.445081 12715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:06:47.438628 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.102.3:45145 (local address 127.0.0.1:34482)
0620 06:06:47.439543 (+ 915us) negotiation.cc:107] Waiting for socket to connect
0620 06:06:47.439559 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 06:06:47.439674 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:06:47.440522 (+ 848us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:06:47.440525 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:06:47.440549 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 06:06:47.440652 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:06:47.440659 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:06:47.441777 (+ 1118us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:06:47.441783 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:06:47.442859 (+ 1076us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:06:47.442870 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:06:47.444217 (+ 1347us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:06:47.444221 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:06:47.444314 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:06:47.444333 (+ 19us) client_negotiation.cc:769] Sending connection context
0620 06:06:47.444402 (+ 69us) client_negotiation.cc:240] Negotiation successful
0620 06:06:47.444459 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":854,"mutex_wait_us":50,"thread_start_us":84,"threads_started":1}
I20260620 06:06:47.445245 12713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:06:47.439257 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.102.1:40869 (local address 127.0.0.1:57218)
0620 06:06:47.440150 (+ 893us) negotiation.cc:107] Waiting for socket to connect
0620 06:06:47.440163 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 06:06:47.440451 (+ 288us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:06:47.440944 (+ 493us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:06:47.440947 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:06:47.440966 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 06:06:47.441042 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:06:47.441048 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:06:47.442405 (+ 1357us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:06:47.442409 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:06:47.443532 (+ 1123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:06:47.443542 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:06:47.444828 (+ 1286us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:06:47.444831 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:06:47.444920 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:06:47.444951 (+ 31us) client_negotiation.cc:769] Sending connection context
0620 06:06:47.445023 (+ 72us) client_negotiation.cc:240] Negotiation successful
0620 06:06:47.445100 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":827,"mutex_wait_us":3,"thread_start_us":90,"threads_started":1}
I20260620 06:06:47.445674 12717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:06:47.439457 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57218 (local address 127.11.102.1:40869)
0620 06:06:47.440141 (+ 684us) server_negotiation.cc:206] Beginning negotiation
0620 06:06:47.440158 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0620 06:06:47.440253 (+ 95us) server_negotiation.cc:407] Connection header received
0620 06:06:47.440617 (+ 364us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:06:47.440624 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:06:47.440808 (+ 184us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:06:47.440954 (+ 146us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 06:06:47.441163 (+ 209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:06:47.442264 (+ 1101us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:06:47.443692 (+ 1428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:06:47.444512 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:06:47.444629 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:06:47.444769 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0620 06:06:47.445399 (+ 630us) server_negotiation.cc:299] Negotiation successful
0620 06:06:47.445489 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":588,"thread_start_us":82,"threads_started":1}
I20260620 06:06:47.448129 12635 tablet_service.cc:1467] Processing CreateTablet for tablet a8334f504a5d49c68248bb140a4dc9f0 (DEFAULT_TABLE table=test-table [id=5737163043c447e7b36abd2d261d612a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 06:06:47.448129 12501 tablet_service.cc:1467] Processing CreateTablet for tablet a8334f504a5d49c68248bb140a4dc9f0 (DEFAULT_TABLE table=test-table [id=5737163043c447e7b36abd2d261d612a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 06:06:47.448388 12367 tablet_service.cc:1467] Processing CreateTablet for tablet a8334f504a5d49c68248bb140a4dc9f0 (DEFAULT_TABLE table=test-table [id=5737163043c447e7b36abd2d261d612a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 06:06:47.449256 12367 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a8334f504a5d49c68248bb140a4dc9f0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 06:06:47.449263 12501 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a8334f504a5d49c68248bb140a4dc9f0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 06:06:47.449263 12635 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a8334f504a5d49c68248bb140a4dc9f0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 06:06:47.455874 12718 tablet_bootstrap.cc:492] T a8334f504a5d49c68248bb140a4dc9f0 P 3f1f48c18638443383a6cac67477eb85: Bootstrap starting.
I20260620 06:06:47.456151 12719 tablet_bootstrap.cc:492] T a8334f504a5d49c68248bb140a4dc9f0 P 1fc1ea4cbd454945817965da6a7e9270: Bootstrap starting.
I20260620 06:06:47.457223 12720 tablet_bootstrap.cc:492] T a8334f504a5d49c68248bb140a4dc9f0 P 5d62c8e649da4ec6ab8e6ee19b2210d4: Bootstrap starting.
I20260620 06:06:47.458424 12719 tablet_bootstrap.cc:654] T a8334f504a5d49c68248bb140a4dc9f0 P 1fc1ea4cbd454945817965da6a7e9270: Neither blocks nor log segments found. Creating new log.
I20260620 06:06:47.458424 12718 tablet_bootstrap.cc:654] T a8334f504a5d49c68248bb140a4dc9f0 P 3f1f48c18638443383a6cac67477eb85: Neither blocks nor log segments found. Creating new log.
I20260620 06:06:47.459237 12720 tablet_bootstrap.cc:654] T a8334f504a5d49c68248bb140a4dc9f0 P 5d62c8e649da4ec6ab8e6ee19b2210d4: Neither blocks nor log segments found. Creating new log.
I20260620 06:06:47.459798 12719 log.cc:824] T a8334f504a5d49c68248bb140a4dc9f0 P 1fc1ea4cbd454945817965da6a7e9270: Log is configured to *not* fsync() on all Append() calls
I20260620 06:06:47.459798 12718 log.cc:824] T a8334f504a5d49c68248bb140a4dc9f0 P 3f1f48c18638443383a6cac67477eb85: Log is configured to *not* fsync() on all Append() calls
I20260620 06:06:47.460034 12720 log.cc:824] T a8334f504a5d49c68248bb140a4dc9f0 P 5d62c8e649da4ec6ab8e6ee19b2210d4: Log is configured to *not* fsync() on all Append() calls
I20260620 06:06:47.462050 12720 tablet_bootstrap.cc:492] T a8334f504a5d49c68248bb140a4dc9f0 P 5d62c8e649da4ec6ab8e6ee19b2210d4: No bootstrap required, opened a new log
I20260620 06:06:47.462077 12719 tablet_bootstrap.cc:492] T a8334f504a5d49c68248bb140a4dc9f0 P 1fc1ea4cbd454945817965da6a7e9270: No bootstrap required, opened a new log
I20260620 06:06:47.462244 12720 ts_tablet_manager.cc:1397] T a8334f504a5d49c68248bb140a4dc9f0 P 5d62c8e649da4ec6ab8e6ee19b2210d4: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260620 06:06:47.462258 12719 ts_tablet_manager.cc:1397] T a8334f504a5d49c68248bb140a4dc9f0 P 1fc1ea4cbd454945817965da6a7e9270: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260620 06:06:47.462423 12718 tablet_bootstrap.cc:492] T a8334f504a5d49c68248bb140a4dc9f0 P 3f1f48c18638443383a6cac67477eb85: No bootstrap required, opened a new log
I20260620 06:06:47.462589 12718 ts_tablet_manager.cc:1397] T a8334f504a5d49c68248bb140a4dc9f0 P 3f1f48c18638443383a6cac67477eb85: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260620 06:06:47.465435 12719 raft_consensus.cc:348] T a8334f504a5d49c68248bb140a4dc9f0 P 1fc1ea4cbd454945817965da6a7e9270 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d62c8e649da4ec6ab8e6ee19b2210d4" member_type: VOTER last_known_addr { host: "127.11.102.2" port: 42273 } } peers { permanent_uuid: "1fc1ea4cbd454945817965da6a7e9270" member_type: VOTER last_known_addr { host: "127.11.102.3" port: 45145 } } peers { permanent_uuid: "3f1f48c18638443383a6cac67477eb85" member_type: VOTER last_known_addr { host: "127.11.102.1" port: 40869 } }
I20260620 06:06:47.465677 12719 raft_consensus.cc:374] T a8334f504a5d49c68248bb140a4dc9f0 P 1fc1ea4cbd454945817965da6a7e9270 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 06:06:47.465728 12719 raft_consensus.cc:729] T a8334f504a5d49c68248bb140a4dc9f0 P 1fc1ea4cbd454945817965da6a7e9270 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1fc1ea4cbd454945817965da6a7e9270, State: Initialized, Role: FOLLOWER
I20260620 06:06:47.466106 12719 consensus_queue.cc:260] T a8334f504a5d49c68248bb140a4dc9f0 P 1fc1ea4cbd454945817965da6a7e9270 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d62c8e649da4ec6ab8e6ee19b2210d4" member_type: VOTER last_known_addr { host: "127.11.102.2" port: 42273 } } peers { permanent_uuid: "1fc1ea4cbd454945817965da6a7e9270" member_type: VOTER last_known_addr { host: "127.11.102.3" port: 45145 } } peers { permanent_uuid: "3f1f48c18638443383a6cac67477eb85" member_type: VOTER last_known_addr { host: "127.11.102.1" port: 40869 } }
I20260620 06:06:47.466162 12718 raft_consensus.cc:348] T a8334f504a5d49c68248bb140a4dc9f0 P 3f1f48c18638443383a6cac67477eb85 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d62c8e649da4ec6ab8e6ee19b2210d4" member_type: VOTER last_known_addr { host: "127.11.102.2" port: 42273 } } peers { permanent_uuid: "1fc1ea4cbd454945817965da6a7e9270" member_type: VOTER last_known_addr { host: "127.11.102.3" port: 45145 } } peers { permanent_uuid: "3f1f48c18638443383a6cac67477eb85" member_type: VOTER last_known_addr { host: "127.11.102.1" port: 40869 } }
I20260620 06:06:47.466121 12720 raft_consensus.cc:348] T a8334f504a5d49c68248bb140a4dc9f0 P 5d62c8e649da4ec6ab8e6ee19b2210d4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d62c8e649da4ec6ab8e6ee19b2210d4" member_type: VOTER last_known_addr { host: "127.11.102.2" port: 42273 } } peers { permanent_uuid: "1fc1ea4cbd454945817965da6a7e9270" member_type: VOTER last_known_addr { host: "127.11.102.3" port: 45145 } } peers { permanent_uuid: "3f1f48c18638443383a6cac67477eb85" member_type: VOTER last_known_addr { host: "127.11.102.1" port: 40869 } }
I20260620 06:06:47.466389 12720 raft_consensus.cc:374] T a8334f504a5d49c68248bb140a4dc9f0 P 5d62c8e649da4ec6ab8e6ee19b2210d4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 06:06:47.466389 12718 raft_consensus.cc:374] T a8334f504a5d49c68248bb140a4dc9f0 P 3f1f48c18638443383a6cac67477eb85 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 06:06:47.466445 12718 raft_consensus.cc:729] T a8334f504a5d49c68248bb140a4dc9f0 P 3f1f48c18638443383a6cac67477eb85 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3f1f48c18638443383a6cac67477eb85, State: Initialized, Role: FOLLOWER
I20260620 06:06:47.466445 12720 raft_consensus.cc:729] T a8334f504a5d49c68248bb140a4dc9f0 P 5d62c8e649da4ec6ab8e6ee19b2210d4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5d62c8e649da4ec6ab8e6ee19b2210d4, State: Initialized, Role: FOLLOWER
I20260620 06:06:47.466869 12701 heartbeater.cc:503] Master 127.11.102.62:39097 was elected leader, sending a full tablet report...
I20260620 06:06:47.466918 12720 consensus_queue.cc:260] T a8334f504a5d49c68248bb140a4dc9f0 P 5d62c8e649da4ec6ab8e6ee19b2210d4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d62c8e649da4ec6ab8e6ee19b2210d4" member_type: VOTER last_known_addr { host: "127.11.102.2" port: 42273 } } peers { permanent_uuid: "1fc1ea4cbd454945817965da6a7e9270" member_type: VOTER last_known_addr { host: "127.11.102.3" port: 45145 } } peers { permanent_uuid: "3f1f48c18638443383a6cac67477eb85" member_type: VOTER last_known_addr { host: "127.11.102.1" port: 40869 } }
I20260620 06:06:47.467177 12719 ts_tablet_manager.cc:1428] T a8334f504a5d49c68248bb140a4dc9f0 P 1fc1ea4cbd454945817965da6a7e9270: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260620 06:06:47.467840 12720 ts_tablet_manager.cc:1428] T a8334f504a5d49c68248bb140a4dc9f0 P 5d62c8e649da4ec6ab8e6ee19b2210d4: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260620 06:06:47.467947 12718 consensus_queue.cc:260] T a8334f504a5d49c68248bb140a4dc9f0 P 3f1f48c18638443383a6cac67477eb85 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d62c8e649da4ec6ab8e6ee19b2210d4" member_type: VOTER last_known_addr { host: "127.11.102.2" port: 42273 } } peers { permanent_uuid: "1fc1ea4cbd454945817965da6a7e9270" member_type: VOTER last_known_addr { host: "127.11.102.3" port: 45145 } } peers { permanent_uuid: "3f1f48c18638443383a6cac67477eb85" member_type: VOTER last_known_addr { host: "127.11.102.1" port: 40869 } }
I20260620 06:06:47.468217 12567 heartbeater.cc:503] Master 127.11.102.62:39097 was elected leader, sending a full tablet report...
I20260620 06:06:47.468598 12433 heartbeater.cc:503] Master 127.11.102.62:39097 was elected leader, sending a full tablet report...
I20260620 06:06:47.468842 12718 ts_tablet_manager.cc:1428] T a8334f504a5d49c68248bb140a4dc9f0 P 3f1f48c18638443383a6cac67477eb85: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
W20260620 06:06:47.469738 12434 tablet.cc:2369] T a8334f504a5d49c68248bb140a4dc9f0 P 3f1f48c18638443383a6cac67477eb85: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 06:06:47.505216 12724 raft_consensus.cc:482] T a8334f504a5d49c68248bb140a4dc9f0 P 1fc1ea4cbd454945817965da6a7e9270 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 06:06:47.505443 12724 raft_consensus.cc:504] T a8334f504a5d49c68248bb140a4dc9f0 P 1fc1ea4cbd454945817965da6a7e9270 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d62c8e649da4ec6ab8e6ee19b2210d4" member_type: VOTER last_known_addr { host: "127.11.102.2" port: 42273 } } peers { permanent_uuid: "1fc1ea4cbd454945817965da6a7e9270" member_type: VOTER last_known_addr { host: "127.11.102.3" port: 45145 } } peers { permanent_uuid: "3f1f48c18638443383a6cac67477eb85" member_type: VOTER last_known_addr { host: "127.11.102.1" port: 40869 } }
I20260620 06:06:47.506682 12724 leader_election.cc:288] T a8334f504a5d49c68248bb140a4dc9f0 P 1fc1ea4cbd454945817965da6a7e9270 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5d62c8e649da4ec6ab8e6ee19b2210d4 (127.11.102.2:42273), 3f1f48c18638443383a6cac67477eb85 (127.11.102.1:40869)
I20260620 06:06:47.510710 12703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:06:47.506939 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.102.2:42273 (local address 127.11.102.3:47501)
0620 06:06:47.507058 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0620 06:06:47.507072 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 06:06:47.507205 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:06:47.507510 (+ 305us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:06:47.507513 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:06:47.507532 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 06:06:47.507692 (+ 160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:06:47.507700 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:06:47.508592 (+ 892us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:06:47.508595 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:06:47.509512 (+ 917us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:06:47.509521 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:06:47.510258 (+ 737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:06:47.510262 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:06:47.510367 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:06:47.510393 (+ 26us) client_negotiation.cc:769] Sending connection context
0620 06:06:47.510479 (+ 86us) client_negotiation.cc:240] Negotiation successful
0620 06:06:47.510559 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":46,"mutex_wait_us":22}
I20260620 06:06:47.510727 12714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:06:47.507063 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.102.3:47501 (local address 127.11.102.2:42273)
0620 06:06:47.507205 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0620 06:06:47.507210 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 06:06:47.507224 (+ 14us) server_negotiation.cc:407] Connection header received
0620 06:06:47.507306 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:06:47.507309 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:06:47.507358 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:06:47.507449 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 06:06:47.507829 (+ 380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:06:47.508478 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:06:47.509639 (+ 1161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:06:47.510146 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:06:47.510236 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:06:47.510273 (+ 37us) server_negotiation.cc:1035] Waiting for connection context
0620 06:06:47.510514 (+ 241us) server_negotiation.cc:299] Negotiation successful
0620 06:06:47.510581 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260620 06:06:47.510978 12727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:06:47.507208 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.102.1:40869 (local address 127.11.102.3:45603)
0620 06:06:47.507567 (+ 359us) negotiation.cc:107] Waiting for socket to connect
0620 06:06:47.507578 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 06:06:47.507716 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:06:47.507972 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:06:47.507975 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:06:47.507992 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 06:06:47.508103 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:06:47.508109 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:06:47.508926 (+ 817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:06:47.508929 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:06:47.509832 (+ 903us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:06:47.509842 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:06:47.510568 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:06:47.510572 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:06:47.510666 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:06:47.510688 (+ 22us) client_negotiation.cc:769] Sending connection context
0620 06:06:47.510768 (+ 80us) client_negotiation.cc:240] Negotiation successful
0620 06:06:47.510835 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":282,"mutex_wait_us":3,"thread_start_us":130,"threads_started":1}
I20260620 06:06:47.511149 12717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:06:47.507339 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.102.3:45603 (local address 127.11.102.1:40869)
0620 06:06:47.507481 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0620 06:06:47.507487 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 06:06:47.507665 (+ 178us) server_negotiation.cc:407] Connection header received
0620 06:06:47.507829 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:06:47.507833 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:06:47.507883 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:06:47.507941 (+ 58us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 06:06:47.508219 (+ 278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:06:47.508813 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:06:47.509955 (+ 1142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:06:47.510450 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:06:47.510554 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:06:47.510596 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0620 06:06:47.510954 (+ 358us) server_negotiation.cc:299] Negotiation successful
0620 06:06:47.511035 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260620 06:06:47.511453 12521 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a8334f504a5d49c68248bb140a4dc9f0" candidate_uuid: "1fc1ea4cbd454945817965da6a7e9270" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5d62c8e649da4ec6ab8e6ee19b2210d4" is_pre_election: true
I20260620 06:06:47.511731 12387 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a8334f504a5d49c68248bb140a4dc9f0" candidate_uuid: "1fc1ea4cbd454945817965da6a7e9270" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3f1f48c18638443383a6cac67477eb85" is_pre_election: true
I20260620 06:06:47.511785 12521 raft_consensus.cc:2454] T a8334f504a5d49c68248bb140a4dc9f0 P 5d62c8e649da4ec6ab8e6ee19b2210d4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1fc1ea4cbd454945817965da6a7e9270 in term 0.
I20260620 06:06:47.511991 12387 raft_consensus.cc:2454] T a8334f504a5d49c68248bb140a4dc9f0 P 3f1f48c18638443383a6cac67477eb85 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1fc1ea4cbd454945817965da6a7e9270 in term 0.
I20260620 06:06:47.512317 12591 leader_election.cc:302] T a8334f504a5d49c68248bb140a4dc9f0 P 1fc1ea4cbd454945817965da6a7e9270 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1fc1ea4cbd454945817965da6a7e9270, 5d62c8e649da4ec6ab8e6ee19b2210d4; no voters:
I20260620 06:06:47.512591 12724 raft_consensus.cc:2781] T a8334f504a5d49c68248bb140a4dc9f0 P 1fc1ea4cbd454945817965da6a7e9270 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 06:06:47.512699 12724 raft_consensus.cc:482] T a8334f504a5d49c68248bb140a4dc9f0 P 1fc1ea4cbd454945817965da6a7e9270 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 06:06:47.512759 12724 raft_consensus.cc:3037] T a8334f504a5d49c68248bb140a4dc9f0 P 1fc1ea4cbd454945817965da6a7e9270 [term 0 FOLLOWER]: Advancing to term 1
I20260620 06:06:47.513826 12724 raft_consensus.cc:504] T a8334f504a5d49c68248bb140a4dc9f0 P 1fc1ea4cbd454945817965da6a7e9270 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d62c8e649da4ec6ab8e6ee19b2210d4" member_type: VOTER last_known_addr { host: "127.11.102.2" port: 42273 } } peers { permanent_uuid: "1fc1ea4cbd454945817965da6a7e9270" member_type: VOTER last_known_addr { host: "127.11.102.3" port: 45145 } } peers { permanent_uuid: "3f1f48c18638443383a6cac67477eb85" member_type: VOTER last_known_addr { host: "127.11.102.1" port: 40869 } }
I20260620 06:06:47.514204 12724 leader_election.cc:288] T a8334f504a5d49c68248bb140a4dc9f0 P 1fc1ea4cbd454945817965da6a7e9270 [CANDIDATE]: Term 1 election: Requested vote from peers 5d62c8e649da4ec6ab8e6ee19b2210d4 (127.11.102.2:42273), 3f1f48c18638443383a6cac67477eb85 (127.11.102.1:40869)
I20260620 06:06:47.514616 12521 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a8334f504a5d49c68248bb140a4dc9f0" candidate_uuid: "1fc1ea4cbd454945817965da6a7e9270" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5d62c8e649da4ec6ab8e6ee19b2210d4"
I20260620 06:06:47.514619 12387 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a8334f504a5d49c68248bb140a4dc9f0" candidate_uuid: "1fc1ea4cbd454945817965da6a7e9270" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3f1f48c18638443383a6cac67477eb85"
I20260620 06:06:47.514721 12521 raft_consensus.cc:3037] T a8334f504a5d49c68248bb140a4dc9f0 P 5d62c8e649da4ec6ab8e6ee19b2210d4 [term 0 FOLLOWER]: Advancing to term 1
I20260620 06:06:47.514750 12387 raft_consensus.cc:3037] T a8334f504a5d49c68248bb140a4dc9f0 P 3f1f48c18638443383a6cac67477eb85 [term 0 FOLLOWER]: Advancing to term 1
I20260620 06:06:47.515875 12387 raft_consensus.cc:2454] T a8334f504a5d49c68248bb140a4dc9f0 P 3f1f48c18638443383a6cac67477eb85 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1fc1ea4cbd454945817965da6a7e9270 in term 1.
I20260620 06:06:47.515858 12521 raft_consensus.cc:2454] T a8334f504a5d49c68248bb140a4dc9f0 P 5d62c8e649da4ec6ab8e6ee19b2210d4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1fc1ea4cbd454945817965da6a7e9270 in term 1.
I20260620 06:06:47.516194 12591 leader_election.cc:302] T a8334f504a5d49c68248bb140a4dc9f0 P 1fc1ea4cbd454945817965da6a7e9270 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1fc1ea4cbd454945817965da6a7e9270, 3f1f48c18638443383a6cac67477eb85; no voters:
I20260620 06:06:47.516392 12724 raft_consensus.cc:2781] T a8334f504a5d49c68248bb140a4dc9f0 P 1fc1ea4cbd454945817965da6a7e9270 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 06:06:47.516618 12724 raft_consensus.cc:686] T a8334f504a5d49c68248bb140a4dc9f0 P 1fc1ea4cbd454945817965da6a7e9270 [term 1 LEADER]: Becoming Leader. State: Replica: 1fc1ea4cbd454945817965da6a7e9270, State: Running, Role: LEADER
I20260620 06:06:47.516939 12724 consensus_queue.cc:237] T a8334f504a5d49c68248bb140a4dc9f0 P 1fc1ea4cbd454945817965da6a7e9270 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d62c8e649da4ec6ab8e6ee19b2210d4" member_type: VOTER last_known_addr { host: "127.11.102.2" port: 42273 } } peers { permanent_uuid: "1fc1ea4cbd454945817965da6a7e9270" member_type: VOTER last_known_addr { host: "127.11.102.3" port: 45145 } } peers { permanent_uuid: "3f1f48c18638443383a6cac67477eb85" member_type: VOTER last_known_addr { host: "127.11.102.1" port: 40869 } }
I20260620 06:06:47.519593 12242 catalog_manager.cc:5495] T a8334f504a5d49c68248bb140a4dc9f0 P 1fc1ea4cbd454945817965da6a7e9270 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1fc1ea4cbd454945817965da6a7e9270 (127.11.102.3). New cstate: current_term: 1 leader_uuid: "1fc1ea4cbd454945817965da6a7e9270" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d62c8e649da4ec6ab8e6ee19b2210d4" member_type: VOTER last_known_addr { host: "127.11.102.2" port: 42273 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1fc1ea4cbd454945817965da6a7e9270" member_type: VOTER last_known_addr { host: "127.11.102.3" port: 45145 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3f1f48c18638443383a6cac67477eb85" member_type: VOTER last_known_addr { host: "127.11.102.1" port: 40869 } health_report { overall_health: UNKNOWN } } }
Jun 20 06:06:47 dist-test-slave-9hm5 krb5kdc[12197](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935606, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.102.1@KRBTEST.COM
I20260620 06:06:47.545692 12717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:06:47.537209 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57230 (local address 127.11.102.1:40869)
0620 06:06:47.537358 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0620 06:06:47.537363 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 06:06:47.537547 (+ 184us) server_negotiation.cc:407] Connection header received
0620 06:06:47.537643 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:06:47.537645 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:06:47.537690 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:06:47.537762 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:06:47.538505 (+ 743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:06:47.538998 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:06:47.539890 (+ 892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:06:47.540096 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:06:47.540180 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:06:47.542549 (+ 2369us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:06:47.542560 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:06:47.542574 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:06:47.542596 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:06:47.544115 (+ 1519us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:06:47.544527 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:06:47.544532 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:06:47.544533 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:06:47.544618 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:06:47.544840 (+ 222us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:06:47.544842 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:06:47.544843 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:06:47.545056 (+ 213us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:06:47.545227 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0620 06:06:47.545384 (+ 157us) server_negotiation.cc:299] Negotiation successful
0620 06:06:47.545523 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
Jun 20 06:06:47 dist-test-slave-9hm5 krb5kdc[12197](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935607, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 20 06:06:47 dist-test-slave-9hm5 krb5kdc[12197](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935607, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.11.102.1@KRBTEST.COM
I20260620 06:06:47.571282 12717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:06:47.563492 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57232 (local address 127.11.102.1:40869)
0620 06:06:47.563697 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0620 06:06:47.563701 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 06:06:47.563715 (+ 14us) server_negotiation.cc:407] Connection header received
0620 06:06:47.563894 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:06:47.563897 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:06:47.563959 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:06:47.564029 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:06:47.564733 (+ 704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:06:47.565215 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:06:47.566002 (+ 787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:06:47.566199 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:06:47.566283 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:06:47.568547 (+ 2264us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:06:47.568550 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:06:47.568552 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:06:47.568574 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:06:47.569796 (+ 1222us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:06:47.570183 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:06:47.570186 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:06:47.570187 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:06:47.570259 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:06:47.570479 (+ 220us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:06:47.570480 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:06:47.570482 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:06:47.570625 (+ 143us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:06:47.570723 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0620 06:06:47.570961 (+ 238us) server_negotiation.cc:299] Negotiation successful
0620 06:06:47.571089 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
Jun 20 06:06:47 dist-test-slave-9hm5 krb5kdc[12197](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935607, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 20 06:06:47 dist-test-slave-9hm5 krb5kdc[12197](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935607, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.102.1@KRBTEST.COM
I20260620 06:06:47.595252 12717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:06:47.587664 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57244 (local address 127.11.102.1:40869)
0620 06:06:47.587807 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0620 06:06:47.587814 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 06:06:47.587925 (+ 111us) server_negotiation.cc:407] Connection header received
0620 06:06:47.588031 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:06:47.588033 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:06:47.588074 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:06:47.588142 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:06:47.588751 (+ 609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:06:47.589247 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:06:47.590039 (+ 792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:06:47.590258 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:06:47.590334 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:06:47.592528 (+ 2194us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:06:47.592531 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:06:47.592534 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:06:47.592555 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:06:47.593751 (+ 1196us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:06:47.594106 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:06:47.594108 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:06:47.594109 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:06:47.594178 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:06:47.594430 (+ 252us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:06:47.594434 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:06:47.594435 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:06:47.594613 (+ 178us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:06:47.594741 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0620 06:06:47.594939 (+ 198us) server_negotiation.cc:299] Negotiation successful
0620 06:06:47.595061 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
W20260620 06:06:47.596603 12347 tablet_service.cc:2908] Rejecting scan request for tablet a8334f504a5d49c68248bb140a4dc9f0: Uninitialized: clean time has not yet been initialized
I20260620 06:06:47.598505 11672 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9HMp43/build/debug/bin/kudu with pid 12304
I20260620 06:06:47.604080 11672 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9HMp43/build/debug/bin/kudu with pid 12438
I20260620 06:06:47.609334 11672 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9HMp43/build/debug/bin/kudu with pid 12572
I20260620 06:06:47.614450 11672 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9HMp43/build/debug/bin/kudu with pid 12210
2026-06-20T06:06:47Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (2264 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781935603120653-11672-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 06:06:47 dist-test-slave-9hm5 krb5kdc[12742](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 06:06:47 dist-test-slave-9hm5 krb5kdc[12742](info): set up 2 sockets
Jun 20 06:06:47 dist-test-slave-9hm5 krb5kdc[12742](info): commencing operation
krb5kdc: starting...
W20260620 06:06:48.647220 11672 mini_kdc.cc:121] Time spent starting KDC: real 1.011s user 0.004s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781935603120653-11672-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781935603120653-11672-0/krb5kdc/test-user.keytab.
Jun 20 06:06:48 dist-test-slave-9hm5 krb5kdc[12742](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935608, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T06:06:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T06:06:48Z Disabled control of system clock
WARNING: no policy specified for kudu/127.11.102.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.102.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.102.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781935603120653-11672-0/krb5kdc/kudu_127.11.102.62.keytab.
Entry for principal kudu/127.11.102.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781935603120653-11672-0/krb5kdc/kudu_127.11.102.62.keytab.
I20260620 06:06:48.760699 11672 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9HMp43/build/debug/bin/kudu
/tmp/dist-test-task9HMp43/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781935603120653-11672-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781935603120653-11672-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781935603120653-11672-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781935603120653-11672-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.102.62:39945
--webserver_interface=127.11.102.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781935603120653-11672-0/krb5kdc/kudu_127.11.102.62.keytab
--principal=kudu/127.11.102.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.102.20:40233
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.102.62:39945
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781935603120653-11672-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781935603120653-11672-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781935603120653-11672-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 06:06:48.866216 12755 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:06:48.866462 12755 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:06:48.866501 12755 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:06:48.870286 12755 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 06:06:48.870355 12755 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260620 06:06:48.870373 12755 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:06:48.870391 12755 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:06:48.870406 12755 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 06:06:48.870421 12755 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 06:06:48.875357 12755 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.102.20:40233
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781935603120653-11672-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781935603120653-11672-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.102.62:39945
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781935603120653-11672-0/krb5kdc/kudu_127.11.102.62.keytab
--principal=kudu/127.11.102.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.102.62:39945
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781935603120653-11672-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.102.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12755
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781935603120653-11672-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 05:28:48 UTC on ccycloud.quasar-fixhms.root.comops.site
build id 4457020
I20260620 06:06:48.876266 12755 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:06:48.877172 12755 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 06:06:48.885248 12760 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:06:48.885320 12761 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:06:48.885381 12755 server_base.cc:1034] running on GCE node
W20260620 06:06:48.885246 12763 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:06:48.886003 12755 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:06:48.887012 12755 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:06:48.888159 12755 hybrid_clock.cc:648] HybridClock initialized: now 1781935608888127 us; error 47 us; skew 500 ppm
Jun 20 06:06:48 dist-test-slave-9hm5 krb5kdc[12742](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935608, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.102.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 06:06:48.890786 12755 init.cc:377] Logged in from keytab as kudu/127.11.102.62@KRBTEST.COM (short username kudu)
I20260620 06:06:48.891678 12755 webserver.cc:460] Webserver started at http://127.11.102.62:35773/ using document root <none> and password file <none>
I20260620 06:06:48.892207 12755 fs_manager.cc:359] Metadata directory not provided
I20260620 06:06:48.892267 12755 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:06:48.892475 12755 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 06:06:48.894364 12755 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781935603120653-11672-0/minicluster-data/master-0/data/instance:
uuid: "6e6f3d203fc348ba9e349e81d952fc19"
format_stamp: "Formatted at 2026-06-20 06:06:48 on dist-test-slave-9hm5"
server_key: "4f8e6a570658f19b7867fc20eecf93e6"
server_key_iv: "bb985c67d56f42e0fe164039aceb42e1"
server_key_version: "encryptionkey@0"
I20260620 06:06:48.894902 12755 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781935603120653-11672-0/minicluster-data/master-0/wal/instance:
uuid: "6e6f3d203fc348ba9e349e81d952fc19"
format_stamp: "Formatted at 2026-06-20 06:06:48 on dist-test-slave-9hm5"
server_key: "4f8e6a570658f19b7867fc20eecf93e6"
server_key_iv: "bb985c67d56f42e0fe164039aceb42e1"
server_key_version: "encryptionkey@0"
I20260620 06:06:48.898339 12755 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260620 06:06:48.900650 12770 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:06:48.901846 12755 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 06:06:48.901966 12755 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781935603120653-11672-0/minicluster-data/master-0/data,/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781935603120653-11672-0/minicluster-data/master-0/wal
uuid: "6e6f3d203fc348ba9e349e81d952fc19"
format_stamp: "Formatted at 2026-06-20 06:06:48 on dist-test-slave-9hm5"
server_key: "4f8e6a570658f19b7867fc20eecf93e6"
server_key_iv: "bb985c67d56f42e0fe164039aceb42e1"
server_key_version: "encryptionkey@0"
I20260620 06:06:48.902063 12755 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781935603120653-11672-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781935603120653-11672-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781935603120653-11672-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:06:48.920722 12755 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:06:48.923593 12755 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:06:48.923822 12755 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:06:48.931375 12755 rpc_server.cc:307] RPC server started. Bound to: 127.11.102.62:39945
I20260620 06:06:48.931394 12832 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.102.62:39945 every 8 connection(s)
I20260620 06:06:48.932308 12755 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781935603120653-11672-0/minicluster-data/master-0/data/info.pb
I20260620 06:06:48.934886 12833 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 06:06:48.935876 11672 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9HMp43/build/debug/bin/kudu as pid 12755
I20260620 06:06:48.935990 11672 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781935603120653-11672-0/minicluster-data/master-0/wal/instance
I20260620 06:06:48.936261 11672 external_mini_cluster.cc:1428] Setting key 65a4407d2c72dbb1524dd60ac4e5b9cc
I20260620 06:06:48.941046 12833 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6e6f3d203fc348ba9e349e81d952fc19: Bootstrap starting.
I20260620 06:06:48.943712 12833 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6e6f3d203fc348ba9e349e81d952fc19: Neither blocks nor log segments found. Creating new log.
I20260620 06:06:48.944558 12833 log.cc:824] T 00000000000000000000000000000000 P 6e6f3d203fc348ba9e349e81d952fc19: Log is configured to *not* fsync() on all Append() calls
Jun 20 06:06:48 dist-test-slave-9hm5 krb5kdc[12742](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935608, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.102.62@KRBTEST.COM
I20260620 06:06:48.946370 12833 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6e6f3d203fc348ba9e349e81d952fc19: No bootstrap required, opened a new log
I20260620 06:06:48.949692 12833 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6e6f3d203fc348ba9e349e81d952fc19 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e6f3d203fc348ba9e349e81d952fc19" member_type: VOTER last_known_addr { host: "127.11.102.62" port: 39945 } }
I20260620 06:06:48.949931 12833 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6e6f3d203fc348ba9e349e81d952fc19 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 06:06:48.950008 12833 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6e6f3d203fc348ba9e349e81d952fc19 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6e6f3d203fc348ba9e349e81d952fc19, State: Initialized, Role: FOLLOWER
I20260620 06:06:48.950347 12836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:06:48.937552 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55900 (local address 127.11.102.62:39945)
0620 06:06:48.937949 (+ 397us) server_negotiation.cc:206] Beginning negotiation
0620 06:06:48.937964 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 06:06:48.937999 (+ 35us) server_negotiation.cc:407] Connection header received
0620 06:06:48.938589 (+ 590us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:06:48.938605 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:06:48.938927 (+ 322us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:06:48.939189 (+ 262us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:06:48.940072 (+ 883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:06:48.941022 (+ 950us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:06:48.942122 (+ 1100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:06:48.942574 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:06:48.942688 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:06:48.945422 (+ 2734us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:06:48.945426 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:06:48.945438 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:06:48.945460 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:06:48.947558 (+ 2098us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:06:48.948076 (+ 518us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:06:48.948081 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:06:48.948085 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:06:48.948199 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:06:48.948468 (+ 269us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:06:48.948470 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:06:48.948471 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:06:48.948806 (+ 335us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:06:48.948963 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0620 06:06:48.949228 (+ 265us) server_negotiation.cc:299] Negotiation successful
0620 06:06:48.949527 (+ 299us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":246,"thread_start_us":124,"threads_started":1}
I20260620 06:06:48.950524 12833 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6e6f3d203fc348ba9e349e81d952fc19 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e6f3d203fc348ba9e349e81d952fc19" member_type: VOTER last_known_addr { host: "127.11.102.62" port: 39945 } }
I20260620 06:06:48.950685 12833 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6e6f3d203fc348ba9e349e81d952fc19 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 06:06:48.950749 12833 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6e6f3d203fc348ba9e349e81d952fc19 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 06:06:48.950843 12833 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6e6f3d203fc348ba9e349e81d952fc19 [term 0 FOLLOWER]: Advancing to term 1
I20260620 06:06:48.951716 12833 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6e6f3d203fc348ba9e349e81d952fc19 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e6f3d203fc348ba9e349e81d952fc19" member_type: VOTER last_known_addr { host: "127.11.102.62" port: 39945 } }
I20260620 06:06:48.952049 12833 leader_election.cc:302] T 00000000000000000000000000000000 P 6e6f3d203fc348ba9e349e81d952fc19 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6e6f3d203fc348ba9e349e81d952fc19; no voters:
I20260620 06:06:48.952381 12833 leader_election.cc:288] T 00000000000000000000000000000000 P 6e6f3d203fc348ba9e349e81d952fc19 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 06:06:48.952519 12838 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6e6f3d203fc348ba9e349e81d952fc19 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 06:06:48.952702 12838 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6e6f3d203fc348ba9e349e81d952fc19 [term 1 LEADER]: Becoming Leader. State: Replica: 6e6f3d203fc348ba9e349e81d952fc19, State: Running, Role: LEADER
I20260620 06:06:48.952975 12838 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6e6f3d203fc348ba9e349e81d952fc19 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e6f3d203fc348ba9e349e81d952fc19" member_type: VOTER last_known_addr { host: "127.11.102.62" port: 39945 } }
I20260620 06:06:48.953557 12833 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6e6f3d203fc348ba9e349e81d952fc19 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 06:06:48.954582 12839 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6e6f3d203fc348ba9e349e81d952fc19 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6e6f3d203fc348ba9e349e81d952fc19" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e6f3d203fc348ba9e349e81d952fc19" member_type: VOTER last_known_addr { host: "127.11.102.62" port: 39945 } } }
I20260620 06:06:48.954706 12839 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6e6f3d203fc348ba9e349e81d952fc19 [sys.catalog]: This master's current role is: LEADER
I20260620 06:06:48.954958 12840 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6e6f3d203fc348ba9e349e81d952fc19 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6e6f3d203fc348ba9e349e81d952fc19. Latest consensus state: current_term: 1 leader_uuid: "6e6f3d203fc348ba9e349e81d952fc19" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e6f3d203fc348ba9e349e81d952fc19" member_type: VOTER last_known_addr { host: "127.11.102.62" port: 39945 } } }
I20260620 06:06:48.955046 12840 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6e6f3d203fc348ba9e349e81d952fc19 [sys.catalog]: This master's current role is: LEADER
I20260620 06:06:48.955139 12847 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 06:06:48.958243 12847 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 06:06:48.963840 12847 catalog_manager.cc:1347] Generated new cluster ID: ae5f47b5a576495aa508faa25ba5ae8b
I20260620 06:06:48.963904 12847 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 06:06:48.984349 12847 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 06:06:48.985194 12847 catalog_manager.cc:1504] Loading token signing keys...
I20260620 06:06:48.992508 12847 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6e6f3d203fc348ba9e349e81d952fc19: Generated new TSK 0
I20260620 06:06:48.993165 12847 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.11.102.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.102.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.102.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781935603120653-11672-0/krb5kdc/kudu_127.11.102.1.keytab.
Entry for principal kudu/127.11.102.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781935603120653-11672-0/krb5kdc/kudu_127.11.102.1.keytab.
I20260620 06:06:49.020534 11672 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9HMp43/build/debug/bin/kudu
/tmp/dist-test-task9HMp43/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781935603120653-11672-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781935603120653-11672-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781935603120653-11672-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781935603120653-11672-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.102.1:0
--local_ip_for_outbound_sockets=127.11.102.1
--webserver_interface=127.11.102.1
--webserver_port=0
--tserver_master_addrs=127.11.102.62:39945
--keytab_file=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781935603120653-11672-0/krb5kdc/kudu_127.11.102.1.keytab
--principal=kudu/127.11.102.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.102.20:40233
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781935603120653-11672-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781935603120653-11672-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781935603120653-11672-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 06:06:49.126825 12859 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:06:49.127056 12859 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:06:49.127132 12859 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:06:49.131117 12859 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:06:49.131170 12859 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:06:49.131247 12859 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 06:06:49.131330 12859 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.102.1
I20260620 06:06:49.136511 12859 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.102.20:40233
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781935603120653-11672-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781935603120653-11672-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781935603120653-11672-0/krb5kdc/kudu_127.11.102.1.keytab
--principal=kudu/127.11.102.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.102.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781935603120653-11672-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.11.102.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.11.102.62:39945
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12859
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.102.1
--log_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781935603120653-11672-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 05:28:48 UTC on ccycloud.quasar-fixhms.root.comops.site
build id 4457020
I20260620 06:06:49.137490 12859 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:06:49.138484 12859 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 06:06:49.147609 12864 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:06:49.147631 12867 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:06:49.147603 12865 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:06:49.147835 12859 server_base.cc:1034] running on GCE node
I20260620 06:06:49.148232 12859 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:06:49.148835 12859 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:06:49.150023 12859 hybrid_clock.cc:648] HybridClock initialized: now 1781935609149990 us; error 46 us; skew 500 ppm
Jun 20 06:06:49 dist-test-slave-9hm5 krb5kdc[12742](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935609, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.102.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 06:06:49.152679 12859 init.cc:377] Logged in from keytab as kudu/127.11.102.1@KRBTEST.COM (short username kudu)
I20260620 06:06:49.153462 12859 webserver.cc:460] Webserver started at http://127.11.102.1:44627/ using document root <none> and password file <none>
I20260620 06:06:49.154057 12859 fs_manager.cc:359] Metadata directory not provided
I20260620 06:06:49.154116 12859 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:06:49.154284 12859 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 06:06:49.156385 12859 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781935603120653-11672-0/minicluster-data/ts-0/data/instance:
uuid: "142bdda3511244afba1b84aa91654fc7"
format_stamp: "Formatted at 2026-06-20 06:06:49 on dist-test-slave-9hm5"
server_key: "cff82145c4e6923332f7bf6ad0959c16"
server_key_iv: "98ee38aabd96bf33866577cc6ead6f92"
server_key_version: "encryptionkey@0"
I20260620 06:06:49.156885 12859 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781935603120653-11672-0/minicluster-data/ts-0/wal/instance:
uuid: "142bdda3511244afba1b84aa91654fc7"
format_stamp: "Formatted at 2026-06-20 06:06:49 on dist-test-slave-9hm5"
server_key: "cff82145c4e6923332f7bf6ad0959c16"
server_key_iv: "98ee38aabd96bf33866577cc6ead6f92"
server_key_version: "encryptionkey@0"
I20260620 06:06:49.160369 12859 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260620 06:06:49.162592 12874 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:06:49.163790 12859 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 06:06:49.163877 12859 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781935603120653-11672-0/minicluster-data/ts-0/data,/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781935603120653-11672-0/minicluster-data/ts-0/wal
uuid: "142bdda3511244afba1b84aa91654fc7"
format_stamp: "Formatted at 2026-06-20 06:06:49 on dist-test-slave-9hm5"
server_key: "cff82145c4e6923332f7bf6ad0959c16"
server_key_iv: "98ee38aabd96bf33866577cc6ead6f92"
server_key_version: "encryptionkey@0"
I20260620 06:06:49.163952 12859 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781935603120653-11672-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781935603120653-11672-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781935603120653-11672-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:06:49.182919 12859 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:06:49.185714 12859 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:06:49.185858 12859 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:06:49.187136 12859 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 06:06:49.187191 12859 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 06:06:49.187256 12859 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 06:06:49.187291 12859 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 06:06:49 dist-test-slave-9hm5 krb5kdc[12742](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935609, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.102.1@KRBTEST.COM for kudu/127.11.102.62@KRBTEST.COM
I20260620 06:06:49.198427 12859 rpc_server.cc:307] RPC server started. Bound to: 127.11.102.1:44353
I20260620 06:06:49.198558 12989 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.102.1:44353 every 8 connection(s)
I20260620 06:06:49.199295 12859 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781935603120653-11672-0/minicluster-data/ts-0/data/info.pb
I20260620 06:06:49.200938 12836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:06:49.188085 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.102.1:48417 (local address 127.11.102.62:39945)
0620 06:06:49.188210 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0620 06:06:49.188213 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 06:06:49.189323 (+ 1110us) server_negotiation.cc:407] Connection header received
0620 06:06:49.190512 (+ 1189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:06:49.190516 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:06:49.190574 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:06:49.190683 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:06:49.192043 (+ 1360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:06:49.192596 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:06:49.193955 (+ 1359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:06:49.194307 (+ 352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:06:49.194412 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:06:49.197359 (+ 2947us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:06:49.197363 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:06:49.197366 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:06:49.197392 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:06:49.198968 (+ 1576us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:06:49.199481 (+ 513us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:06:49.199485 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:06:49.199487 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:06:49.199565 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:06:49.199966 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:06:49.199969 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:06:49.199971 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:06:49.200198 (+ 227us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:06:49.200277 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0620 06:06:49.200718 (+ 441us) server_negotiation.cc:299] Negotiation successful
0620 06:06:49.200823 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260620 06:06:49.201674 12883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:06:49.188625 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.102.62:39945 (local address 127.11.102.1:48417)
0620 06:06:49.189131 (+ 506us) negotiation.cc:107] Waiting for socket to connect
0620 06:06:49.189185 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0620 06:06:49.190272 (+ 1087us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:06:49.190849 (+ 577us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:06:49.190857 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:06:49.191259 (+ 402us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:06:49.191887 (+ 628us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:06:49.191906 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:06:49.192720 (+ 814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:06:49.192725 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:06:49.193831 (+ 1106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:06:49.193839 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:06:49.194455 (+ 616us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:06:49.194459 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:06:49.194641 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:06:49.195255 (+ 614us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:06:49.195280 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:06:49.197199 (+ 1919us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:06:49.199078 (+ 1879us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:06:49.199084 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:06:49.199098 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:06:49.199391 (+ 293us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:06:49.199673 (+ 282us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:06:49.199677 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:06:49.199682 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:06:49.199848 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:06:49.200305 (+ 457us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:06:49.200309 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:06:49.200515 (+ 206us) client_negotiation.cc:769] Sending connection context
0620 06:06:49.200675 (+ 160us) client_negotiation.cc:240] Negotiation successful
0620 06:06:49.200904 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":322,"thread_start_us":139,"threads_started":1}
I20260620 06:06:49.202875 12990 heartbeater.cc:347] Connected to a master server at 127.11.102.62:39945
I20260620 06:06:49.203040 12990 heartbeater.cc:464] Registering TS with master...
I20260620 06:06:49.203490 12990 heartbeater.cc:511] Master 127.11.102.62:39945 requested a full tablet report, sending...
I20260620 06:06:49.205001 12786 ts_manager.cc:194] Registered new tserver with Master: 142bdda3511244afba1b84aa91654fc7 (127.11.102.1:44353)
I20260620 06:06:49.206076 11672 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9HMp43/build/debug/bin/kudu as pid 12859
I20260620 06:06:49.206182 11672 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781935603120653-11672-0/minicluster-data/ts-0/wal/instance
I20260620 06:06:49.206462 11672 external_mini_cluster.cc:1428] Setting key e5d20b6feeccb81918dd9540fabfb63c
I20260620 06:06:49.206485 12786 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.102.1@KRBTEST.COM'} at 127.11.102.1:48417
I20260620 06:06:49.212240 12836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:06:49.204962 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.102.1:45961 (local address 127.11.102.62:39945)
0620 06:06:49.205081 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0620 06:06:49.205085 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 06:06:49.205256 (+ 171us) server_negotiation.cc:407] Connection header received
0620 06:06:49.205370 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:06:49.205378 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:06:49.205417 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:06:49.205480 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:06:49.205994 (+ 514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:06:49.206709 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:06:49.207891 (+ 1182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:06:49.208106 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:06:49.208162 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:06:49.209256 (+ 1094us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:06:49.209262 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:06:49.209267 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:06:49.209295 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:06:49.210626 (+ 1331us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:06:49.211094 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:06:49.211096 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:06:49.211097 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:06:49.211162 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:06:49.211447 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:06:49.211449 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:06:49.211450 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:06:49.211589 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:06:49.211745 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0620 06:06:49.212055 (+ 310us) server_negotiation.cc:299] Negotiation successful
0620 06:06:49.212137 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260620 06:06:49.212353 12996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:06:49.204905 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.102.62:39945 (local address 127.11.102.1:45961)
0620 06:06:49.205163 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0620 06:06:49.205186 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0620 06:06:49.205279 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:06:49.205519 (+ 240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:06:49.205522 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:06:49.205710 (+ 188us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:06:49.205876 (+ 166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:06:49.205884 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:06:49.206858 (+ 974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:06:49.206861 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:06:49.207762 (+ 901us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:06:49.207772 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:06:49.208271 (+ 499us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:06:49.208273 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:06:49.208334 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:06:49.208804 (+ 470us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:06:49.208822 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:06:49.209114 (+ 292us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:06:49.210734 (+ 1620us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:06:49.210740 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:06:49.210745 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:06:49.211000 (+ 255us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:06:49.211251 (+ 251us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:06:49.211257 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:06:49.211262 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:06:49.211360 (+ 98us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:06:49.211764 (+ 404us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:06:49.211768 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:06:49.211878 (+ 110us) client_negotiation.cc:769] Sending connection context
0620 06:06:49.211995 (+ 117us) client_negotiation.cc:240] Negotiation successful
0620 06:06:49.212138 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":97,"threads_started":1}
WARNING: no policy specified for kudu/127.11.102.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.102.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.102.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781935603120653-11672-0/krb5kdc/kudu_127.11.102.2.keytab.
Entry for principal kudu/127.11.102.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781935603120653-11672-0/krb5kdc/kudu_127.11.102.2.keytab.
I20260620 06:06:49.232609 11672 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9HMp43/build/debug/bin/kudu
/tmp/dist-test-task9HMp43/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781935603120653-11672-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781935603120653-11672-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781935603120653-11672-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781935603120653-11672-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.102.2:0
--local_ip_for_outbound_sockets=127.11.102.2
--webserver_interface=127.11.102.2
--webserver_port=0
--tserver_master_addrs=127.11.102.62:39945
--keytab_file=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781935603120653-11672-0/krb5kdc/kudu_127.11.102.2.keytab
--principal=kudu/127.11.102.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.102.20:40233
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781935603120653-11672-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781935603120653-11672-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781935603120653-11672-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 06:06:49.339505 12999 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:06:49.339819 12999 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:06:49.339900 12999 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:06:49.347424 12999 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:06:49.347510 12999 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:06:49.347573 12999 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 06:06:49.347666 12999 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.102.2
I20260620 06:06:49.352795 12999 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.102.20:40233
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781935603120653-11672-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781935603120653-11672-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781935603120653-11672-0/krb5kdc/kudu_127.11.102.2.keytab
--principal=kudu/127.11.102.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.102.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781935603120653-11672-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.11.102.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.11.102.62:39945
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12999
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.102.2
--log_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781935603120653-11672-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 05:28:48 UTC on ccycloud.quasar-fixhms.root.comops.site
build id 4457020
I20260620 06:06:49.353878 12999 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:06:49.354950 12999 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 06:06:49.363909 13004 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:06:49.363932 12999 server_base.cc:1034] running on GCE node
W20260620 06:06:49.363940 13005 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:06:49.363900 13007 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:06:49.364527 12999 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:06:49.365088 12999 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:06:49.366274 12999 hybrid_clock.cc:648] HybridClock initialized: now 1781935609366261 us; error 25 us; skew 500 ppm
Jun 20 06:06:49 dist-test-slave-9hm5 krb5kdc[12742](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935609, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.102.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 06:06:49.368968 12999 init.cc:377] Logged in from keytab as kudu/127.11.102.2@KRBTEST.COM (short username kudu)
I20260620 06:06:49.369758 12999 webserver.cc:460] Webserver started at http://127.11.102.2:33947/ using document root <none> and password file <none>
I20260620 06:06:49.370267 12999 fs_manager.cc:359] Metadata directory not provided
I20260620 06:06:49.370322 12999 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:06:49.370469 12999 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 06:06:49.372321 12999 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781935603120653-11672-0/minicluster-data/ts-1/data/instance:
uuid: "bebfb222b0064148be502c27d76fa23f"
format_stamp: "Formatted at 2026-06-20 06:06:49 on dist-test-slave-9hm5"
server_key: "7e4ecaadbad0a51a10c85c4bd218f612"
server_key_iv: "8296f57e45002fcd71aece19730e25b5"
server_key_version: "encryptionkey@0"
I20260620 06:06:49.372773 12999 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781935603120653-11672-0/minicluster-data/ts-1/wal/instance:
uuid: "bebfb222b0064148be502c27d76fa23f"
format_stamp: "Formatted at 2026-06-20 06:06:49 on dist-test-slave-9hm5"
server_key: "7e4ecaadbad0a51a10c85c4bd218f612"
server_key_iv: "8296f57e45002fcd71aece19730e25b5"
server_key_version: "encryptionkey@0"
I20260620 06:06:49.375994 12999 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 06:06:49.378170 13014 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:06:49.379317 12999 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 06:06:49.379405 12999 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781935603120653-11672-0/minicluster-data/ts-1/data,/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781935603120653-11672-0/minicluster-data/ts-1/wal
uuid: "bebfb222b0064148be502c27d76fa23f"
format_stamp: "Formatted at 2026-06-20 06:06:49 on dist-test-slave-9hm5"
server_key: "7e4ecaadbad0a51a10c85c4bd218f612"
server_key_iv: "8296f57e45002fcd71aece19730e25b5"
server_key_version: "encryptionkey@0"
I20260620 06:06:49.379529 12999 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781935603120653-11672-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781935603120653-11672-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781935603120653-11672-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:06:49.393414 12999 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:06:49.396458 12999 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:06:49.396664 12999 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:06:49.397967 12999 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 06:06:49.398033 12999 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 06:06:49.398099 12999 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 06:06:49.398147 12999 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 06:06:49 dist-test-slave-9hm5 krb5kdc[12742](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935609, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.102.2@KRBTEST.COM for kudu/127.11.102.62@KRBTEST.COM
I20260620 06:06:49.408258 12999 rpc_server.cc:307] RPC server started. Bound to: 127.11.102.2:33501
I20260620 06:06:49.408828 13129 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.102.2:33501 every 8 connection(s)
I20260620 06:06:49.409205 12999 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781935603120653-11672-0/minicluster-data/ts-1/data/info.pb
I20260620 06:06:49.412104 12836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:06:49.398754 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.102.2:53847 (local address 127.11.102.62:39945)
0620 06:06:49.398894 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0620 06:06:49.398899 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 06:06:49.399745 (+ 846us) server_negotiation.cc:407] Connection header received
0620 06:06:49.400848 (+ 1103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:06:49.400855 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:06:49.400920 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:06:49.401034 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:06:49.402726 (+ 1692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:06:49.403663 (+ 937us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:06:49.405514 (+ 1851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:06:49.405728 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:06:49.405811 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:06:49.408788 (+ 2977us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:06:49.408791 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:06:49.408793 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:06:49.408816 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:06:49.410416 (+ 1600us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:06:49.410860 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:06:49.410863 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:06:49.410864 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:06:49.410936 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:06:49.411196 (+ 260us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:06:49.411197 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:06:49.411198 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:06:49.411344 (+ 146us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:06:49.411411 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0620 06:06:49.411901 (+ 490us) server_negotiation.cc:299] Negotiation successful
0620 06:06:49.411986 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260620 06:06:49.412850 13023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:06:49.399126 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.102.62:39945 (local address 127.11.102.2:53847)
0620 06:06:49.399551 (+ 425us) negotiation.cc:107] Waiting for socket to connect
0620 06:06:49.399635 (+ 84us) client_negotiation.cc:174] Beginning negotiation
0620 06:06:49.400612 (+ 977us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:06:49.401366 (+ 754us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:06:49.401381 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:06:49.401831 (+ 450us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:06:49.402555 (+ 724us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:06:49.402577 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:06:49.403801 (+ 1224us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:06:49.403808 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:06:49.405368 (+ 1560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:06:49.405381 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:06:49.405880 (+ 499us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:06:49.405887 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:06:49.406125 (+ 238us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:06:49.406776 (+ 651us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:06:49.406802 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:06:49.408622 (+ 1820us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:06:49.410512 (+ 1890us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:06:49.410516 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:06:49.410525 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:06:49.410774 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:06:49.411016 (+ 242us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:06:49.411018 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:06:49.411019 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:06:49.411136 (+ 117us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:06:49.411423 (+ 287us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:06:49.411427 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:06:49.411688 (+ 261us) client_negotiation.cc:769] Sending connection context
0620 06:06:49.411855 (+ 167us) client_negotiation.cc:240] Negotiation successful
0620 06:06:49.412085 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":244,"thread_start_us":127,"threads_started":1}
I20260620 06:06:49.414027 13130 heartbeater.cc:347] Connected to a master server at 127.11.102.62:39945
I20260620 06:06:49.414197 13130 heartbeater.cc:464] Registering TS with master...
I20260620 06:06:49.414676 13130 heartbeater.cc:511] Master 127.11.102.62:39945 requested a full tablet report, sending...
I20260620 06:06:49.415746 12786 ts_manager.cc:194] Registered new tserver with Master: bebfb222b0064148be502c27d76fa23f (127.11.102.2:33501)
I20260620 06:06:49.416481 12786 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.102.2@KRBTEST.COM'} at 127.11.102.2:53847
I20260620 06:06:49.418226 11672 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9HMp43/build/debug/bin/kudu as pid 12999
I20260620 06:06:49.418349 11672 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781935603120653-11672-0/minicluster-data/ts-1/wal/instance
I20260620 06:06:49.418643 11672 external_mini_cluster.cc:1428] Setting key 5464e08790fa8f303ae27661f832dc38
I20260620 06:06:49.425462 13136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:06:49.416056 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.102.62:39945 (local address 127.11.102.2:48999)
0620 06:06:49.416323 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0620 06:06:49.416337 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 06:06:49.416440 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:06:49.416659 (+ 219us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:06:49.416662 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:06:49.416900 (+ 238us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:06:49.417016 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:06:49.417027 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:06:49.418042 (+ 1015us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:06:49.418060 (+ 18us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:06:49.418892 (+ 832us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:06:49.418902 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:06:49.419375 (+ 473us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:06:49.419378 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:06:49.419466 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:06:49.421452 (+ 1986us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:06:49.421464 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:06:49.421771 (+ 307us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:06:49.423757 (+ 1986us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:06:49.423762 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:06:49.423764 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:06:49.423997 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:06:49.424313 (+ 316us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:06:49.424318 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:06:49.424320 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:06:49.424423 (+ 103us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:06:49.424880 (+ 457us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:06:49.424884 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:06:49.424996 (+ 112us) client_negotiation.cc:769] Sending connection context
0620 06:06:49.425114 (+ 118us) client_negotiation.cc:240] Negotiation successful
0620 06:06:49.425269 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":109,"threads_started":1}
I20260620 06:06:49.425444 12836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:06:49.416128 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.102.2:48999 (local address 127.11.102.62:39945)
0620 06:06:49.416252 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0620 06:06:49.416257 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 06:06:49.416417 (+ 160us) server_negotiation.cc:407] Connection header received
0620 06:06:49.416516 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:06:49.416518 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:06:49.416553 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:06:49.416647 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:06:49.417147 (+ 500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:06:49.417880 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:06:49.419016 (+ 1136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:06:49.419284 (+ 268us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:06:49.419363 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:06:49.421920 (+ 2557us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:06:49.421926 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:06:49.421932 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:06:49.421965 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:06:49.423563 (+ 1598us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:06:49.424115 (+ 552us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:06:49.424119 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:06:49.424121 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:06:49.424211 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:06:49.424537 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:06:49.424540 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:06:49.424542 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:06:49.424763 (+ 221us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:06:49.424880 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0620 06:06:49.425171 (+ 291us) server_negotiation.cc:299] Negotiation successful
0620 06:06:49.425281 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
WARNING: no policy specified for kudu/127.11.102.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.102.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.102.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781935603120653-11672-0/krb5kdc/kudu_127.11.102.3.keytab.
Entry for principal kudu/127.11.102.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781935603120653-11672-0/krb5kdc/kudu_127.11.102.3.keytab.
I20260620 06:06:49.444550 11672 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9HMp43/build/debug/bin/kudu
/tmp/dist-test-task9HMp43/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781935603120653-11672-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781935603120653-11672-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781935603120653-11672-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781935603120653-11672-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.102.3:0
--local_ip_for_outbound_sockets=127.11.102.3
--webserver_interface=127.11.102.3
--webserver_port=0
--tserver_master_addrs=127.11.102.62:39945
--keytab_file=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781935603120653-11672-0/krb5kdc/kudu_127.11.102.3.keytab
--principal=kudu/127.11.102.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.102.20:40233
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781935603120653-11672-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781935603120653-11672-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781935603120653-11672-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 06:06:49.569114 13139 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:06:49.569404 13139 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:06:49.569484 13139 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:06:49.573653 13139 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:06:49.573729 13139 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:06:49.573807 13139 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 06:06:49.573884 13139 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.102.3
I20260620 06:06:49.579166 13139 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.102.20:40233
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781935603120653-11672-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781935603120653-11672-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781935603120653-11672-0/krb5kdc/kudu_127.11.102.3.keytab
--principal=kudu/127.11.102.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.102.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781935603120653-11672-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.11.102.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.11.102.62:39945
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13139
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.102.3
--log_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781935603120653-11672-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 05:28:48 UTC on ccycloud.quasar-fixhms.root.comops.site
build id 4457020
I20260620 06:06:49.580231 13139 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:06:49.581208 13139 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 06:06:49.590662 13147 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:06:49.590639 13145 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:06:49.590646 13139 server_base.cc:1034] running on GCE node
W20260620 06:06:49.590633 13144 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:06:49.591234 13139 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:06:49.591858 13139 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:06:49.593029 13139 hybrid_clock.cc:648] HybridClock initialized: now 1781935609593007 us; error 33 us; skew 500 ppm
Jun 20 06:06:49 dist-test-slave-9hm5 krb5kdc[12742](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935609, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.102.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 06:06:49.595772 13139 init.cc:377] Logged in from keytab as kudu/127.11.102.3@KRBTEST.COM (short username kudu)
I20260620 06:06:49.596580 13139 webserver.cc:460] Webserver started at http://127.11.102.3:45991/ using document root <none> and password file <none>
I20260620 06:06:49.597105 13139 fs_manager.cc:359] Metadata directory not provided
I20260620 06:06:49.597163 13139 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:06:49.597373 13139 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 06:06:49.599287 13139 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781935603120653-11672-0/minicluster-data/ts-2/data/instance:
uuid: "a77868c9753b4ced97fa3d6f1146e0cf"
format_stamp: "Formatted at 2026-06-20 06:06:49 on dist-test-slave-9hm5"
server_key: "67207716b87bafe9092bc508970b620a"
server_key_iv: "97a05e67ff1e7dd2df752ca02ac946ea"
server_key_version: "encryptionkey@0"
I20260620 06:06:49.599848 13139 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781935603120653-11672-0/minicluster-data/ts-2/wal/instance:
uuid: "a77868c9753b4ced97fa3d6f1146e0cf"
format_stamp: "Formatted at 2026-06-20 06:06:49 on dist-test-slave-9hm5"
server_key: "67207716b87bafe9092bc508970b620a"
server_key_iv: "97a05e67ff1e7dd2df752ca02ac946ea"
server_key_version: "encryptionkey@0"
I20260620 06:06:49.603384 13139 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260620 06:06:49.605715 13154 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:06:49.606935 13139 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 06:06:49.607035 13139 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781935603120653-11672-0/minicluster-data/ts-2/data,/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781935603120653-11672-0/minicluster-data/ts-2/wal
uuid: "a77868c9753b4ced97fa3d6f1146e0cf"
format_stamp: "Formatted at 2026-06-20 06:06:49 on dist-test-slave-9hm5"
server_key: "67207716b87bafe9092bc508970b620a"
server_key_iv: "97a05e67ff1e7dd2df752ca02ac946ea"
server_key_version: "encryptionkey@0"
I20260620 06:06:49.607156 13139 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781935603120653-11672-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781935603120653-11672-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781935603120653-11672-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:06:49.620275 13139 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:06:49.622931 13139 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:06:49.623114 13139 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:06:49.624518 13139 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 06:06:49.624567 13139 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 06:06:49.624603 13139 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 06:06:49.624639 13139 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 06:06:49 dist-test-slave-9hm5 krb5kdc[12742](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935609, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.102.3@KRBTEST.COM for kudu/127.11.102.62@KRBTEST.COM
I20260620 06:06:49.635679 13139 rpc_server.cc:307] RPC server started. Bound to: 127.11.102.3:42987
I20260620 06:06:49.635758 13269 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.102.3:42987 every 8 connection(s)
I20260620 06:06:49.636745 13139 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781935603120653-11672-0/minicluster-data/ts-2/data/info.pb
I20260620 06:06:49.637548 12836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:06:49.625261 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.102.3:50545 (local address 127.11.102.62:39945)
0620 06:06:49.625425 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0620 06:06:49.625433 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 06:06:49.626598 (+ 1165us) server_negotiation.cc:407] Connection header received
0620 06:06:49.627681 (+ 1083us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:06:49.627684 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:06:49.627740 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:06:49.627841 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:06:49.629034 (+ 1193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:06:49.629555 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:06:49.630874 (+ 1319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:06:49.631155 (+ 281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:06:49.631271 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:06:49.633614 (+ 2343us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:06:49.633618 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:06:49.633620 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:06:49.633651 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:06:49.635378 (+ 1727us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:06:49.636001 (+ 623us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:06:49.636005 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:06:49.636007 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:06:49.636086 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:06:49.636417 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:06:49.636420 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:06:49.636422 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:06:49.636628 (+ 206us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:06:49.636730 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0620 06:06:49.637253 (+ 523us) server_negotiation.cc:299] Negotiation successful
0620 06:06:49.637380 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260620 06:06:49.638334 13163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:06:49.626027 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.102.62:39945 (local address 127.11.102.3:50545)
0620 06:06:49.626438 (+ 411us) negotiation.cc:107] Waiting for socket to connect
0620 06:06:49.626483 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0620 06:06:49.627449 (+ 966us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:06:49.628009 (+ 560us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:06:49.628016 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:06:49.628345 (+ 329us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:06:49.628895 (+ 550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:06:49.628907 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:06:49.629678 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:06:49.629683 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:06:49.630749 (+ 1066us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:06:49.630758 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:06:49.631316 (+ 558us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:06:49.631320 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:06:49.631478 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:06:49.632046 (+ 568us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:06:49.632060 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:06:49.633457 (+ 1397us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:06:49.635515 (+ 2058us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:06:49.635522 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:06:49.635537 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:06:49.635874 (+ 337us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:06:49.636178 (+ 304us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:06:49.636180 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:06:49.636182 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:06:49.636305 (+ 123us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:06:49.636757 (+ 452us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:06:49.636763 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:06:49.637014 (+ 251us) client_negotiation.cc:769] Sending connection context
0620 06:06:49.637195 (+ 181us) client_negotiation.cc:240] Negotiation successful
0620 06:06:49.637516 (+ 321us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":232,"thread_start_us":123,"threads_started":1}
I20260620 06:06:49.639428 13270 heartbeater.cc:347] Connected to a master server at 127.11.102.62:39945
I20260620 06:06:49.639606 13270 heartbeater.cc:464] Registering TS with master...
I20260620 06:06:49.640071 13270 heartbeater.cc:511] Master 127.11.102.62:39945 requested a full tablet report, sending...
I20260620 06:06:49.640424 11672 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9HMp43/build/debug/bin/kudu as pid 13139
I20260620 06:06:49.640530 11672 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781935603120653-11672-0/minicluster-data/ts-2/wal/instance
I20260620 06:06:49.640800 11672 external_mini_cluster.cc:1428] Setting key 4d0a5d3c925185c32301ef22bd214820
I20260620 06:06:49.641665 12787 ts_manager.cc:194] Registered new tserver with Master: a77868c9753b4ced97fa3d6f1146e0cf (127.11.102.3:42987)
I20260620 06:06:49.642450 12787 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.102.3@KRBTEST.COM'} at 127.11.102.3:50545
I20260620 06:06:49.642906 11672 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 06:06:49.651001 12836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:06:49.641671 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.102.3:55257 (local address 127.11.102.62:39945)
0620 06:06:49.641802 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0620 06:06:49.641807 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 06:06:49.642134 (+ 327us) server_negotiation.cc:407] Connection header received
0620 06:06:49.642230 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:06:49.642232 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:06:49.642272 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:06:49.642350 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:06:49.642909 (+ 559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:06:49.643667 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:06:49.645022 (+ 1355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:06:49.645388 (+ 366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:06:49.645486 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:06:49.646787 (+ 1301us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:06:49.646791 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:06:49.646794 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:06:49.646816 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:06:49.649166 (+ 2350us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:06:49.649565 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:06:49.649570 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:06:49.649572 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:06:49.649651 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:06:49.649961 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:06:49.649967 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:06:49.649970 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:06:49.650216 (+ 246us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:06:49.650315 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0620 06:06:49.650710 (+ 395us) server_negotiation.cc:299] Negotiation successful
0620 06:06:49.650831 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260620 06:06:49.651481 13276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:06:49.641724 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.102.62:39945 (local address 127.11.102.3:55257)
0620 06:06:49.642036 (+ 312us) negotiation.cc:107] Waiting for socket to connect
0620 06:06:49.642051 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 06:06:49.642158 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:06:49.642377 (+ 219us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:06:49.642380 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:06:49.642570 (+ 190us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:06:49.642684 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:06:49.642693 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:06:49.643809 (+ 1116us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:06:49.643815 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:06:49.644843 (+ 1028us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:06:49.644855 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:06:49.645501 (+ 646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:06:49.645505 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:06:49.645600 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:06:49.646122 (+ 522us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:06:49.646135 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:06:49.646644 (+ 509us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:06:49.649272 (+ 2628us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:06:49.649275 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:06:49.649277 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:06:49.649470 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:06:49.649760 (+ 290us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:06:49.649763 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:06:49.649765 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:06:49.649851 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:06:49.650364 (+ 513us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:06:49.650369 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:06:49.650480 (+ 111us) client_negotiation.cc:769] Sending connection context
0620 06:06:49.651133 (+ 653us) client_negotiation.cc:240] Negotiation successful
0620 06:06:49.651277 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":236,"mutex_wait_us":1,"thread_start_us":101,"threads_started":1}
Jun 20 06:06:49 dist-test-slave-9hm5 krb5kdc[12742](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935609, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 20 06:06:49 dist-test-slave-9hm5 krb5kdc[12742](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935609, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.11.102.62@KRBTEST.COM
I20260620 06:06:49.672379 12836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:06:49.664558 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55912 (local address 127.11.102.62:39945)
0620 06:06:49.664687 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0620 06:06:49.664690 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 06:06:49.664702 (+ 12us) server_negotiation.cc:407] Connection header received
0620 06:06:49.664740 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:06:49.664742 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:06:49.664779 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:06:49.664849 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:06:49.665542 (+ 693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:06:49.666035 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:06:49.666825 (+ 790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:06:49.667038 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:06:49.667119 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:06:49.669592 (+ 2473us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:06:49.669595 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:06:49.669597 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:06:49.669619 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:06:49.670814 (+ 1195us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:06:49.671226 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:06:49.671228 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:06:49.671229 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:06:49.671297 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:06:49.671514 (+ 217us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:06:49.671516 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:06:49.671517 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:06:49.671746 (+ 229us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:06:49.671847 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0620 06:06:49.672084 (+ 237us) server_negotiation.cc:299] Negotiation successful
0620 06:06:49.672171 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260620 06:06:49.675109 12786 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55912:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 06:06:49.678287 12786 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 06:06:49.691375 13289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:06:49.686313 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60878 (local address 127.11.102.1:44353)
0620 06:06:49.686702 (+ 389us) server_negotiation.cc:206] Beginning negotiation
0620 06:06:49.686718 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0620 06:06:49.686744 (+ 26us) server_negotiation.cc:407] Connection header received
0620 06:06:49.687004 (+ 260us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:06:49.687010 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:06:49.687226 (+ 216us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:06:49.687381 (+ 155us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 06:06:49.687881 (+ 500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:06:49.688854 (+ 973us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:06:49.690208 (+ 1354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:06:49.690732 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:06:49.690853 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:06:49.691028 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0620 06:06:49.691135 (+ 107us) server_negotiation.cc:299] Negotiation successful
0620 06:06:49.691224 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":295,"thread_start_us":73,"threads_started":1}
I20260620 06:06:49.691359 13288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:06:49.686183 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.102.1:44353 (local address 127.0.0.1:60878)
0620 06:06:49.686617 (+ 434us) negotiation.cc:107] Waiting for socket to connect
0620 06:06:49.686668 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0620 06:06:49.686797 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:06:49.687549 (+ 752us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:06:49.687555 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:06:49.687581 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 06:06:49.687746 (+ 165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:06:49.687755 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:06:49.688953 (+ 1198us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:06:49.688957 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:06:49.690074 (+ 1117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:06:49.690086 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:06:49.690851 (+ 765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:06:49.690854 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:06:49.690957 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:06:49.690992 (+ 35us) client_negotiation.cc:769] Sending connection context
0620 06:06:49.691108 (+ 116us) client_negotiation.cc:240] Negotiation successful
0620 06:06:49.691221 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":259,"thread_start_us":76,"threads_started":1}
I20260620 06:06:49.693054 13290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:06:49.686892 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.102.3:42987 (local address 127.0.0.1:56992)
0620 06:06:49.687665 (+ 773us) negotiation.cc:107] Waiting for socket to connect
0620 06:06:49.687676 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 06:06:49.687759 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:06:49.688188 (+ 429us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:06:49.688190 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:06:49.688206 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 06:06:49.688292 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:06:49.688299 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:06:49.689710 (+ 1411us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:06:49.689714 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:06:49.691092 (+ 1378us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:06:49.691099 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:06:49.692444 (+ 1345us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:06:49.692447 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:06:49.692691 (+ 244us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:06:49.692773 (+ 82us) client_negotiation.cc:769] Sending connection context
0620 06:06:49.692845 (+ 72us) client_negotiation.cc:240] Negotiation successful
0620 06:06:49.692917 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":689,"mutex_wait_us":194,"thread_start_us":97,"threads_started":1}
I20260620 06:06:49.693140 13291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:06:49.686981 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46098 (local address 127.11.102.2:33501)
0620 06:06:49.687372 (+ 391us) server_negotiation.cc:206] Beginning negotiation
0620 06:06:49.687382 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 06:06:49.687672 (+ 290us) server_negotiation.cc:407] Connection header received
0620 06:06:49.687872 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:06:49.687878 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:06:49.688066 (+ 188us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:06:49.688213 (+ 147us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 06:06:49.688568 (+ 355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:06:49.689640 (+ 1072us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:06:49.691226 (+ 1586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:06:49.692091 (+ 865us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:06:49.692172 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:06:49.692327 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0620 06:06:49.692904 (+ 577us) server_negotiation.cc:299] Negotiation successful
0620 06:06:49.692990 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":304,"thread_start_us":108,"threads_started":1}
I20260620 06:06:49.693491 13287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:06:49.686260 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.102.2:33501 (local address 127.0.0.1:46098)
0620 06:06:49.687549 (+ 1289us) negotiation.cc:107] Waiting for socket to connect
0620 06:06:49.687564 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 06:06:49.687686 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:06:49.688336 (+ 650us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:06:49.688339 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:06:49.688354 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 06:06:49.688439 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:06:49.688444 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:06:49.689749 (+ 1305us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:06:49.689752 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:06:49.691089 (+ 1337us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:06:49.691099 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:06:49.692424 (+ 1325us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:06:49.692428 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:06:49.692693 (+ 265us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:06:49.692774 (+ 81us) client_negotiation.cc:769] Sending connection context
0620 06:06:49.693300 (+ 526us) client_negotiation.cc:240] Negotiation successful
0620 06:06:49.693366 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1227,"mutex_wait_us":201,"thread_start_us":81,"threads_started":1}
I20260620 06:06:49.693501 13292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:06:49.687274 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56992 (local address 127.11.102.3:42987)
0620 06:06:49.687572 (+ 298us) server_negotiation.cc:206] Beginning negotiation
0620 06:06:49.687583 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 06:06:49.687738 (+ 155us) server_negotiation.cc:407] Connection header received
0620 06:06:49.687902 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:06:49.687907 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:06:49.688060 (+ 153us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:06:49.688214 (+ 154us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 06:06:49.688412 (+ 198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:06:49.689577 (+ 1165us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:06:49.691226 (+ 1649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:06:49.692175 (+ 949us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:06:49.692264 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:06:49.692399 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0620 06:06:49.693287 (+ 888us) server_negotiation.cc:299] Negotiation successful
0620 06:06:49.693375 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":207,"thread_start_us":94,"threads_started":1}
I20260620 06:06:49.694190 12924 tablet_service.cc:1467] Processing CreateTablet for tablet 4be434645ce240faaedc7ad1494249a8 (DEFAULT_TABLE table=test-table [id=7ae4807bc2094ae2a78a4a649f1c9d0f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 06:06:49.695298 12924 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4be434645ce240faaedc7ad1494249a8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 06:06:49.695714 13064 tablet_service.cc:1467] Processing CreateTablet for tablet 4be434645ce240faaedc7ad1494249a8 (DEFAULT_TABLE table=test-table [id=7ae4807bc2094ae2a78a4a649f1c9d0f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 06:06:49.696142 13204 tablet_service.cc:1467] Processing CreateTablet for tablet 4be434645ce240faaedc7ad1494249a8 (DEFAULT_TABLE table=test-table [id=7ae4807bc2094ae2a78a4a649f1c9d0f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 06:06:49.696753 13064 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4be434645ce240faaedc7ad1494249a8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 06:06:49.697086 13204 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4be434645ce240faaedc7ad1494249a8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 06:06:49.702944 13293 tablet_bootstrap.cc:492] T 4be434645ce240faaedc7ad1494249a8 P bebfb222b0064148be502c27d76fa23f: Bootstrap starting.
I20260620 06:06:49.704181 13294 tablet_bootstrap.cc:492] T 4be434645ce240faaedc7ad1494249a8 P 142bdda3511244afba1b84aa91654fc7: Bootstrap starting.
I20260620 06:06:49.704180 13295 tablet_bootstrap.cc:492] T 4be434645ce240faaedc7ad1494249a8 P a77868c9753b4ced97fa3d6f1146e0cf: Bootstrap starting.
I20260620 06:06:49.705521 13293 tablet_bootstrap.cc:654] T 4be434645ce240faaedc7ad1494249a8 P bebfb222b0064148be502c27d76fa23f: Neither blocks nor log segments found. Creating new log.
I20260620 06:06:49.705982 13295 tablet_bootstrap.cc:654] T 4be434645ce240faaedc7ad1494249a8 P a77868c9753b4ced97fa3d6f1146e0cf: Neither blocks nor log segments found. Creating new log.
I20260620 06:06:49.706107 13294 tablet_bootstrap.cc:654] T 4be434645ce240faaedc7ad1494249a8 P 142bdda3511244afba1b84aa91654fc7: Neither blocks nor log segments found. Creating new log.
I20260620 06:06:49.706404 13293 log.cc:824] T 4be434645ce240faaedc7ad1494249a8 P bebfb222b0064148be502c27d76fa23f: Log is configured to *not* fsync() on all Append() calls
I20260620 06:06:49.706810 13295 log.cc:824] T 4be434645ce240faaedc7ad1494249a8 P a77868c9753b4ced97fa3d6f1146e0cf: Log is configured to *not* fsync() on all Append() calls
I20260620 06:06:49.706887 13294 log.cc:824] T 4be434645ce240faaedc7ad1494249a8 P 142bdda3511244afba1b84aa91654fc7: Log is configured to *not* fsync() on all Append() calls
I20260620 06:06:49.708551 13294 tablet_bootstrap.cc:492] T 4be434645ce240faaedc7ad1494249a8 P 142bdda3511244afba1b84aa91654fc7: No bootstrap required, opened a new log
I20260620 06:06:49.708750 13295 tablet_bootstrap.cc:492] T 4be434645ce240faaedc7ad1494249a8 P a77868c9753b4ced97fa3d6f1146e0cf: No bootstrap required, opened a new log
I20260620 06:06:49.708750 13293 tablet_bootstrap.cc:492] T 4be434645ce240faaedc7ad1494249a8 P bebfb222b0064148be502c27d76fa23f: No bootstrap required, opened a new log
I20260620 06:06:49.708786 13294 ts_tablet_manager.cc:1397] T 4be434645ce240faaedc7ad1494249a8 P 142bdda3511244afba1b84aa91654fc7: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260620 06:06:49.708920 13295 ts_tablet_manager.cc:1397] T 4be434645ce240faaedc7ad1494249a8 P a77868c9753b4ced97fa3d6f1146e0cf: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260620 06:06:49.708920 13293 ts_tablet_manager.cc:1397] T 4be434645ce240faaedc7ad1494249a8 P bebfb222b0064148be502c27d76fa23f: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260620 06:06:49.711861 13294 raft_consensus.cc:348] T 4be434645ce240faaedc7ad1494249a8 P 142bdda3511244afba1b84aa91654fc7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "142bdda3511244afba1b84aa91654fc7" member_type: VOTER last_known_addr { host: "127.11.102.1" port: 44353 } } peers { permanent_uuid: "bebfb222b0064148be502c27d76fa23f" member_type: VOTER last_known_addr { host: "127.11.102.2" port: 33501 } } peers { permanent_uuid: "a77868c9753b4ced97fa3d6f1146e0cf" member_type: VOTER last_known_addr { host: "127.11.102.3" port: 42987 } }
I20260620 06:06:49.712095 13294 raft_consensus.cc:374] T 4be434645ce240faaedc7ad1494249a8 P 142bdda3511244afba1b84aa91654fc7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 06:06:49.712145 13294 raft_consensus.cc:729] T 4be434645ce240faaedc7ad1494249a8 P 142bdda3511244afba1b84aa91654fc7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 142bdda3511244afba1b84aa91654fc7, State: Initialized, Role: FOLLOWER
I20260620 06:06:49.712564 13294 consensus_queue.cc:260] T 4be434645ce240faaedc7ad1494249a8 P 142bdda3511244afba1b84aa91654fc7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "142bdda3511244afba1b84aa91654fc7" member_type: VOTER last_known_addr { host: "127.11.102.1" port: 44353 } } peers { permanent_uuid: "bebfb222b0064148be502c27d76fa23f" member_type: VOTER last_known_addr { host: "127.11.102.2" port: 33501 } } peers { permanent_uuid: "a77868c9753b4ced97fa3d6f1146e0cf" member_type: VOTER last_known_addr { host: "127.11.102.3" port: 42987 } }
I20260620 06:06:49.712713 13295 raft_consensus.cc:348] T 4be434645ce240faaedc7ad1494249a8 P a77868c9753b4ced97fa3d6f1146e0cf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "142bdda3511244afba1b84aa91654fc7" member_type: VOTER last_known_addr { host: "127.11.102.1" port: 44353 } } peers { permanent_uuid: "bebfb222b0064148be502c27d76fa23f" member_type: VOTER last_known_addr { host: "127.11.102.2" port: 33501 } } peers { permanent_uuid: "a77868c9753b4ced97fa3d6f1146e0cf" member_type: VOTER last_known_addr { host: "127.11.102.3" port: 42987 } }
I20260620 06:06:49.712713 13293 raft_consensus.cc:348] T 4be434645ce240faaedc7ad1494249a8 P bebfb222b0064148be502c27d76fa23f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "142bdda3511244afba1b84aa91654fc7" member_type: VOTER last_known_addr { host: "127.11.102.1" port: 44353 } } peers { permanent_uuid: "bebfb222b0064148be502c27d76fa23f" member_type: VOTER last_known_addr { host: "127.11.102.2" port: 33501 } } peers { permanent_uuid: "a77868c9753b4ced97fa3d6f1146e0cf" member_type: VOTER last_known_addr { host: "127.11.102.3" port: 42987 } }
I20260620 06:06:49.712940 13295 raft_consensus.cc:374] T 4be434645ce240faaedc7ad1494249a8 P a77868c9753b4ced97fa3d6f1146e0cf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 06:06:49.712953 13293 raft_consensus.cc:374] T 4be434645ce240faaedc7ad1494249a8 P bebfb222b0064148be502c27d76fa23f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 06:06:49.713004 13293 raft_consensus.cc:729] T 4be434645ce240faaedc7ad1494249a8 P bebfb222b0064148be502c27d76fa23f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bebfb222b0064148be502c27d76fa23f, State: Initialized, Role: FOLLOWER
I20260620 06:06:49.713004 13295 raft_consensus.cc:729] T 4be434645ce240faaedc7ad1494249a8 P a77868c9753b4ced97fa3d6f1146e0cf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a77868c9753b4ced97fa3d6f1146e0cf, State: Initialized, Role: FOLLOWER
I20260620 06:06:49.713397 13295 consensus_queue.cc:260] T 4be434645ce240faaedc7ad1494249a8 P a77868c9753b4ced97fa3d6f1146e0cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "142bdda3511244afba1b84aa91654fc7" member_type: VOTER last_known_addr { host: "127.11.102.1" port: 44353 } } peers { permanent_uuid: "bebfb222b0064148be502c27d76fa23f" member_type: VOTER last_known_addr { host: "127.11.102.2" port: 33501 } } peers { permanent_uuid: "a77868c9753b4ced97fa3d6f1146e0cf" member_type: VOTER last_known_addr { host: "127.11.102.3" port: 42987 } }
I20260620 06:06:49.713564 13294 ts_tablet_manager.cc:1428] T 4be434645ce240faaedc7ad1494249a8 P 142bdda3511244afba1b84aa91654fc7: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260620 06:06:49.713956 12990 heartbeater.cc:503] Master 127.11.102.62:39945 was elected leader, sending a full tablet report...
I20260620 06:06:49.714322 13295 ts_tablet_manager.cc:1428] T 4be434645ce240faaedc7ad1494249a8 P a77868c9753b4ced97fa3d6f1146e0cf: Time spent starting tablet: real 0.005s user 0.002s sys 0.005s
I20260620 06:06:49.714664 13270 heartbeater.cc:503] Master 127.11.102.62:39945 was elected leader, sending a full tablet report...
I20260620 06:06:49.713712 13293 consensus_queue.cc:260] T 4be434645ce240faaedc7ad1494249a8 P bebfb222b0064148be502c27d76fa23f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "142bdda3511244afba1b84aa91654fc7" member_type: VOTER last_known_addr { host: "127.11.102.1" port: 44353 } } peers { permanent_uuid: "bebfb222b0064148be502c27d76fa23f" member_type: VOTER last_known_addr { host: "127.11.102.2" port: 33501 } } peers { permanent_uuid: "a77868c9753b4ced97fa3d6f1146e0cf" member_type: VOTER last_known_addr { host: "127.11.102.3" port: 42987 } }
I20260620 06:06:49.715525 13130 heartbeater.cc:503] Master 127.11.102.62:39945 was elected leader, sending a full tablet report...
I20260620 06:06:49.715780 13293 ts_tablet_manager.cc:1428] T 4be434645ce240faaedc7ad1494249a8 P bebfb222b0064148be502c27d76fa23f: Time spent starting tablet: real 0.007s user 0.003s sys 0.003s
I20260620 06:06:49.774557 13300 raft_consensus.cc:482] T 4be434645ce240faaedc7ad1494249a8 P a77868c9753b4ced97fa3d6f1146e0cf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 06:06:49.774753 13300 raft_consensus.cc:504] T 4be434645ce240faaedc7ad1494249a8 P a77868c9753b4ced97fa3d6f1146e0cf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "142bdda3511244afba1b84aa91654fc7" member_type: VOTER last_known_addr { host: "127.11.102.1" port: 44353 } } peers { permanent_uuid: "bebfb222b0064148be502c27d76fa23f" member_type: VOTER last_known_addr { host: "127.11.102.2" port: 33501 } } peers { permanent_uuid: "a77868c9753b4ced97fa3d6f1146e0cf" member_type: VOTER last_known_addr { host: "127.11.102.3" port: 42987 } }
I20260620 06:06:49.776033 13300 leader_election.cc:288] T 4be434645ce240faaedc7ad1494249a8 P a77868c9753b4ced97fa3d6f1146e0cf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 142bdda3511244afba1b84aa91654fc7 (127.11.102.1:44353), bebfb222b0064148be502c27d76fa23f (127.11.102.2:33501)
I20260620 06:06:49.779554 13163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:06:49.776220 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.102.1:44353 (local address 127.11.102.3:57357)
0620 06:06:49.776333 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0620 06:06:49.776350 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 06:06:49.776457 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:06:49.776726 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:06:49.776729 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:06:49.776746 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 06:06:49.776869 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:06:49.776875 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:06:49.777576 (+ 701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:06:49.777583 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:06:49.778371 (+ 788us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:06:49.778379 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:06:49.779097 (+ 718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:06:49.779101 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:06:49.779232 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:06:49.779249 (+ 17us) client_negotiation.cc:769] Sending connection context
0620 06:06:49.779347 (+ 98us) client_negotiation.cc:240] Negotiation successful
0620 06:06:49.779418 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32,"spinlock_wait_cycles":25472}
I20260620 06:06:49.779587 13289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:06:49.776326 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.102.3:57357 (local address 127.11.102.1:44353)
0620 06:06:49.776457 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0620 06:06:49.776464 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 06:06:49.776483 (+ 19us) server_negotiation.cc:407] Connection header received
0620 06:06:49.776548 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:06:49.776551 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:06:49.776604 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:06:49.776705 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 06:06:49.776976 (+ 271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:06:49.777473 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:06:49.778493 (+ 1020us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:06:49.778978 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:06:49.779097 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:06:49.779157 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0620 06:06:49.779393 (+ 236us) server_negotiation.cc:299] Negotiation successful
0620 06:06:49.779466 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260620 06:06:49.780278 13302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:06:49.776348 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.102.2:33501 (local address 127.11.102.3:38771)
0620 06:06:49.776653 (+ 305us) negotiation.cc:107] Waiting for socket to connect
0620 06:06:49.776666 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 06:06:49.776806 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:06:49.777047 (+ 241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:06:49.777049 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:06:49.777063 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 06:06:49.777511 (+ 448us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:06:49.777517 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:06:49.778215 (+ 698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:06:49.778220 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:06:49.778981 (+ 761us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:06:49.778991 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:06:49.779963 (+ 972us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:06:49.779969 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:06:49.780057 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:06:49.780073 (+ 16us) client_negotiation.cc:769] Sending connection context
0620 06:06:49.780133 (+ 60us) client_negotiation.cc:240] Negotiation successful
0620 06:06:49.780179 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":247,"spinlock_wait_cycles":768,"thread_start_us":121,"threads_started":1}
I20260620 06:06:49.780251 12944 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4be434645ce240faaedc7ad1494249a8" candidate_uuid: "a77868c9753b4ced97fa3d6f1146e0cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "142bdda3511244afba1b84aa91654fc7" is_pre_election: true
I20260620 06:06:49.780367 13291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:06:49.776542 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.102.3:38771 (local address 127.11.102.2:33501)
0620 06:06:49.776656 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0620 06:06:49.776662 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 06:06:49.776745 (+ 83us) server_negotiation.cc:407] Connection header received
0620 06:06:49.776916 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:06:49.776919 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:06:49.776956 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:06:49.777037 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 06:06:49.777621 (+ 584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:06:49.778109 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:06:49.779123 (+ 1014us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:06:49.779734 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:06:49.779861 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:06:49.779908 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0620 06:06:49.780176 (+ 268us) server_negotiation.cc:299] Negotiation successful
0620 06:06:49.780248 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260620 06:06:49.780563 12944 raft_consensus.cc:2454] T 4be434645ce240faaedc7ad1494249a8 P 142bdda3511244afba1b84aa91654fc7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a77868c9753b4ced97fa3d6f1146e0cf in term 0.
I20260620 06:06:49.781059 13158 leader_election.cc:302] T 4be434645ce240faaedc7ad1494249a8 P a77868c9753b4ced97fa3d6f1146e0cf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 142bdda3511244afba1b84aa91654fc7, a77868c9753b4ced97fa3d6f1146e0cf; no voters:
I20260620 06:06:49.781064 13084 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4be434645ce240faaedc7ad1494249a8" candidate_uuid: "a77868c9753b4ced97fa3d6f1146e0cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bebfb222b0064148be502c27d76fa23f" is_pre_election: true
I20260620 06:06:49.781349 13300 raft_consensus.cc:2781] T 4be434645ce240faaedc7ad1494249a8 P a77868c9753b4ced97fa3d6f1146e0cf [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 06:06:49.781328 13084 raft_consensus.cc:2454] T 4be434645ce240faaedc7ad1494249a8 P bebfb222b0064148be502c27d76fa23f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a77868c9753b4ced97fa3d6f1146e0cf in term 0.
I20260620 06:06:49.781430 13300 raft_consensus.cc:482] T 4be434645ce240faaedc7ad1494249a8 P a77868c9753b4ced97fa3d6f1146e0cf [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 06:06:49.781461 13300 raft_consensus.cc:3037] T 4be434645ce240faaedc7ad1494249a8 P a77868c9753b4ced97fa3d6f1146e0cf [term 0 FOLLOWER]: Advancing to term 1
I20260620 06:06:49.782693 13300 raft_consensus.cc:504] T 4be434645ce240faaedc7ad1494249a8 P a77868c9753b4ced97fa3d6f1146e0cf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "142bdda3511244afba1b84aa91654fc7" member_type: VOTER last_known_addr { host: "127.11.102.1" port: 44353 } } peers { permanent_uuid: "bebfb222b0064148be502c27d76fa23f" member_type: VOTER last_known_addr { host: "127.11.102.2" port: 33501 } } peers { permanent_uuid: "a77868c9753b4ced97fa3d6f1146e0cf" member_type: VOTER last_known_addr { host: "127.11.102.3" port: 42987 } }
I20260620 06:06:49.783030 13300 leader_election.cc:288] T 4be434645ce240faaedc7ad1494249a8 P a77868c9753b4ced97fa3d6f1146e0cf [CANDIDATE]: Term 1 election: Requested vote from peers 142bdda3511244afba1b84aa91654fc7 (127.11.102.1:44353), bebfb222b0064148be502c27d76fa23f (127.11.102.2:33501)
I20260620 06:06:49.783322 12944 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4be434645ce240faaedc7ad1494249a8" candidate_uuid: "a77868c9753b4ced97fa3d6f1146e0cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "142bdda3511244afba1b84aa91654fc7"
I20260620 06:06:49.783468 12944 raft_consensus.cc:3037] T 4be434645ce240faaedc7ad1494249a8 P 142bdda3511244afba1b84aa91654fc7 [term 0 FOLLOWER]: Advancing to term 1
I20260620 06:06:49.783468 13084 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4be434645ce240faaedc7ad1494249a8" candidate_uuid: "a77868c9753b4ced97fa3d6f1146e0cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bebfb222b0064148be502c27d76fa23f"
I20260620 06:06:49.783601 13084 raft_consensus.cc:3037] T 4be434645ce240faaedc7ad1494249a8 P bebfb222b0064148be502c27d76fa23f [term 0 FOLLOWER]: Advancing to term 1
I20260620 06:06:49.784451 12944 raft_consensus.cc:2454] T 4be434645ce240faaedc7ad1494249a8 P 142bdda3511244afba1b84aa91654fc7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a77868c9753b4ced97fa3d6f1146e0cf in term 1.
I20260620 06:06:49.784736 13158 leader_election.cc:302] T 4be434645ce240faaedc7ad1494249a8 P a77868c9753b4ced97fa3d6f1146e0cf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 142bdda3511244afba1b84aa91654fc7, a77868c9753b4ced97fa3d6f1146e0cf; no voters:
I20260620 06:06:49.784948 13300 raft_consensus.cc:2781] T 4be434645ce240faaedc7ad1494249a8 P a77868c9753b4ced97fa3d6f1146e0cf [term 1 FOLLOWER]: Leader election won for term 1
I20260620 06:06:49.785111 13084 raft_consensus.cc:2454] T 4be434645ce240faaedc7ad1494249a8 P bebfb222b0064148be502c27d76fa23f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a77868c9753b4ced97fa3d6f1146e0cf in term 1.
I20260620 06:06:49.785176 13300 raft_consensus.cc:686] T 4be434645ce240faaedc7ad1494249a8 P a77868c9753b4ced97fa3d6f1146e0cf [term 1 LEADER]: Becoming Leader. State: Replica: a77868c9753b4ced97fa3d6f1146e0cf, State: Running, Role: LEADER
I20260620 06:06:49.785538 13300 consensus_queue.cc:237] T 4be434645ce240faaedc7ad1494249a8 P a77868c9753b4ced97fa3d6f1146e0cf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "142bdda3511244afba1b84aa91654fc7" member_type: VOTER last_known_addr { host: "127.11.102.1" port: 44353 } } peers { permanent_uuid: "bebfb222b0064148be502c27d76fa23f" member_type: VOTER last_known_addr { host: "127.11.102.2" port: 33501 } } peers { permanent_uuid: "a77868c9753b4ced97fa3d6f1146e0cf" member_type: VOTER last_known_addr { host: "127.11.102.3" port: 42987 } }
I20260620 06:06:49.788540 12787 catalog_manager.cc:5495] T 4be434645ce240faaedc7ad1494249a8 P a77868c9753b4ced97fa3d6f1146e0cf reported cstate change: term changed from 0 to 1, leader changed from <none> to a77868c9753b4ced97fa3d6f1146e0cf (127.11.102.3). New cstate: current_term: 1 leader_uuid: "a77868c9753b4ced97fa3d6f1146e0cf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "142bdda3511244afba1b84aa91654fc7" member_type: VOTER last_known_addr { host: "127.11.102.1" port: 44353 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bebfb222b0064148be502c27d76fa23f" member_type: VOTER last_known_addr { host: "127.11.102.2" port: 33501 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a77868c9753b4ced97fa3d6f1146e0cf" member_type: VOTER last_known_addr { host: "127.11.102.3" port: 42987 } health_report { overall_health: HEALTHY } } }
I20260620 06:06:49.816275 13292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:06:49.812746 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57000 (local address 127.11.102.3:42987)
0620 06:06:49.812874 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0620 06:06:49.812877 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 06:06:49.812890 (+ 13us) server_negotiation.cc:407] Connection header received
0620 06:06:49.813052 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:06:49.813054 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:06:49.813104 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:06:49.813185 (+ 81us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 06:06:49.813407 (+ 222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:06:49.814008 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:06:49.814897 (+ 889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:06:49.815107 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:06:49.815182 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:06:49.815486 (+ 304us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 06:06:49.815892 (+ 406us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 06:06:49.815981 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0620 06:06:49.816099 (+ 118us) server_negotiation.cc:299] Negotiation successful
0620 06:06:49.816147 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260620 06:06:49.822178 12944 raft_consensus.cc:1261] T 4be434645ce240faaedc7ad1494249a8 P 142bdda3511244afba1b84aa91654fc7 [term 1 FOLLOWER]: Refusing update from remote peer a77868c9753b4ced97fa3d6f1146e0cf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 06:06:49.822388 13084 raft_consensus.cc:1261] T 4be434645ce240faaedc7ad1494249a8 P bebfb222b0064148be502c27d76fa23f [term 1 FOLLOWER]: Refusing update from remote peer a77868c9753b4ced97fa3d6f1146e0cf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 06:06:49.822926 13303 consensus_queue.cc:1035] T 4be434645ce240faaedc7ad1494249a8 P a77868c9753b4ced97fa3d6f1146e0cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "142bdda3511244afba1b84aa91654fc7" member_type: VOTER last_known_addr { host: "127.11.102.1" port: 44353 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 06:06:49.823153 13300 consensus_queue.cc:1035] T 4be434645ce240faaedc7ad1494249a8 P a77868c9753b4ced97fa3d6f1146e0cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "bebfb222b0064148be502c27d76fa23f" member_type: VOTER last_known_addr { host: "127.11.102.2" port: 33501 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 06:06:49.832122 13307 mvcc.cc:205] Tried to move back new op lower bound from 7298808257823989760 to 7298808257684856832. Current Snapshot: MvccSnapshot[applied={T|T < 7298808257823989760}]
I20260620 06:06:49.841348 12786 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55912:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 06:06:49.841488 12786 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55912:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 06:06:49.843809 12786 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6e6f3d203fc348ba9e349e81d952fc19: Sending DeleteTablet for 3 replicas of tablet 4be434645ce240faaedc7ad1494249a8
I20260620 06:06:49.844552 12924 tablet_service.cc:1514] Processing DeleteTablet for tablet 4be434645ce240faaedc7ad1494249a8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 06:06:49 UTC) from {username='kudu', principal='kudu/127.11.102.62@KRBTEST.COM'} at 127.0.0.1:60878
I20260620 06:06:49.844595 13204 tablet_service.cc:1514] Processing DeleteTablet for tablet 4be434645ce240faaedc7ad1494249a8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 06:06:49 UTC) from {username='kudu', principal='kudu/127.11.102.62@KRBTEST.COM'} at 127.0.0.1:56992
I20260620 06:06:49.845397 13316 tablet_replica.cc:330] T 4be434645ce240faaedc7ad1494249a8 P 142bdda3511244afba1b84aa91654fc7: stopping tablet replica
I20260620 06:06:49.845448 13064 tablet_service.cc:1514] Processing DeleteTablet for tablet 4be434645ce240faaedc7ad1494249a8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 06:06:49 UTC) from {username='kudu', principal='kudu/127.11.102.62@KRBTEST.COM'} at 127.0.0.1:46098
I20260620 06:06:49.845749 13316 raft_consensus.cc:2229] T 4be434645ce240faaedc7ad1494249a8 P 142bdda3511244afba1b84aa91654fc7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 06:06:49.845954 13318 tablet_replica.cc:330] T 4be434645ce240faaedc7ad1494249a8 P bebfb222b0064148be502c27d76fa23f: stopping tablet replica
I20260620 06:06:49.845980 13317 tablet_replica.cc:330] T 4be434645ce240faaedc7ad1494249a8 P a77868c9753b4ced97fa3d6f1146e0cf: stopping tablet replica
I20260620 06:06:49.846236 13317 raft_consensus.cc:2229] T 4be434645ce240faaedc7ad1494249a8 P a77868c9753b4ced97fa3d6f1146e0cf [term 1 LEADER]: Raft consensus shutting down.
I20260620 06:06:49.846346 13318 raft_consensus.cc:2229] T 4be434645ce240faaedc7ad1494249a8 P bebfb222b0064148be502c27d76fa23f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 06:06:49.846604 13317 raft_consensus.cc:2258] T 4be434645ce240faaedc7ad1494249a8 P a77868c9753b4ced97fa3d6f1146e0cf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 06:06:49.846743 13318 raft_consensus.cc:2258] T 4be434645ce240faaedc7ad1494249a8 P bebfb222b0064148be502c27d76fa23f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 06:06:49.846726 12786 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55912:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 06:06:49.847141 12786 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 06:06:49.848100 13317 ts_tablet_manager.cc:1905] T 4be434645ce240faaedc7ad1494249a8 P a77868c9753b4ced97fa3d6f1146e0cf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 06:06:49.848538 13318 ts_tablet_manager.cc:1905] T 4be434645ce240faaedc7ad1494249a8 P bebfb222b0064148be502c27d76fa23f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 06:06:49.848915 13316 raft_consensus.cc:2258] T 4be434645ce240faaedc7ad1494249a8 P 142bdda3511244afba1b84aa91654fc7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 06:06:49.850240 13316 ts_tablet_manager.cc:1905] T 4be434645ce240faaedc7ad1494249a8 P 142bdda3511244afba1b84aa91654fc7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 06:06:49.850703 13317 ts_tablet_manager.cc:1918] T 4be434645ce240faaedc7ad1494249a8 P a77868c9753b4ced97fa3d6f1146e0cf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 06:06:49.850778 13317 log.cc:1196] T 4be434645ce240faaedc7ad1494249a8 P a77868c9753b4ced97fa3d6f1146e0cf: Deleting WAL directory at /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781935603120653-11672-0/minicluster-data/ts-2/wal/wals/4be434645ce240faaedc7ad1494249a8
I20260620 06:06:49.851078 13317 ts_tablet_manager.cc:1939] T 4be434645ce240faaedc7ad1494249a8 P a77868c9753b4ced97fa3d6f1146e0cf: Deleting consensus metadata
I20260620 06:06:49.852150 12773 catalog_manager.cc:4841] TS a77868c9753b4ced97fa3d6f1146e0cf (127.11.102.3:42987): tablet 4be434645ce240faaedc7ad1494249a8 (table test-table [id=7ae4807bc2094ae2a78a4a649f1c9d0f]) successfully deleted
I20260620 06:06:49.853044 12924 tablet_service.cc:1467] Processing CreateTablet for tablet 03e1ead220b94c319ab7aa71cc62e9c0 (DEFAULT_TABLE table=test-table [id=b1000c9283134c26913891dbb0ac09f1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 06:06:49.853318 12924 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 03e1ead220b94c319ab7aa71cc62e9c0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 06:06:49.853292 13064 tablet_service.cc:1467] Processing CreateTablet for tablet 03e1ead220b94c319ab7aa71cc62e9c0 (DEFAULT_TABLE table=test-table [id=b1000c9283134c26913891dbb0ac09f1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 06:06:49.853533 13064 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 03e1ead220b94c319ab7aa71cc62e9c0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 06:06:49.855324 13318 ts_tablet_manager.cc:1918] T 4be434645ce240faaedc7ad1494249a8 P bebfb222b0064148be502c27d76fa23f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 06:06:49.855415 13318 log.cc:1196] T 4be434645ce240faaedc7ad1494249a8 P bebfb222b0064148be502c27d76fa23f: Deleting WAL directory at /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781935603120653-11672-0/minicluster-data/ts-1/wal/wals/4be434645ce240faaedc7ad1494249a8
I20260620 06:06:49.855547 13293 tablet_bootstrap.cc:492] T 03e1ead220b94c319ab7aa71cc62e9c0 P bebfb222b0064148be502c27d76fa23f: Bootstrap starting.
I20260620 06:06:49.855746 13318 ts_tablet_manager.cc:1939] T 4be434645ce240faaedc7ad1494249a8 P bebfb222b0064148be502c27d76fa23f: Deleting consensus metadata
I20260620 06:06:49.855765 13204 tablet_service.cc:1467] Processing CreateTablet for tablet 03e1ead220b94c319ab7aa71cc62e9c0 (DEFAULT_TABLE table=test-table [id=b1000c9283134c26913891dbb0ac09f1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 06:06:49.856000 13204 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 03e1ead220b94c319ab7aa71cc62e9c0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 06:06:49.856515 13293 tablet_bootstrap.cc:654] T 03e1ead220b94c319ab7aa71cc62e9c0 P bebfb222b0064148be502c27d76fa23f: Neither blocks nor log segments found. Creating new log.
I20260620 06:06:49.857342 12773 catalog_manager.cc:4841] TS bebfb222b0064148be502c27d76fa23f (127.11.102.2:33501): tablet 4be434645ce240faaedc7ad1494249a8 (table test-table [id=7ae4807bc2094ae2a78a4a649f1c9d0f]) successfully deleted
I20260620 06:06:49.857757 13293 tablet_bootstrap.cc:492] T 03e1ead220b94c319ab7aa71cc62e9c0 P bebfb222b0064148be502c27d76fa23f: No bootstrap required, opened a new log
I20260620 06:06:49.857841 13293 ts_tablet_manager.cc:1397] T 03e1ead220b94c319ab7aa71cc62e9c0 P bebfb222b0064148be502c27d76fa23f: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260620 06:06:49.858251 13295 tablet_bootstrap.cc:492] T 03e1ead220b94c319ab7aa71cc62e9c0 P a77868c9753b4ced97fa3d6f1146e0cf: Bootstrap starting.
I20260620 06:06:49.858273 13293 raft_consensus.cc:348] T 03e1ead220b94c319ab7aa71cc62e9c0 P bebfb222b0064148be502c27d76fa23f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "142bdda3511244afba1b84aa91654fc7" member_type: VOTER last_known_addr { host: "127.11.102.1" port: 44353 } } peers { permanent_uuid: "bebfb222b0064148be502c27d76fa23f" member_type: VOTER last_known_addr { host: "127.11.102.2" port: 33501 } } peers { permanent_uuid: "a77868c9753b4ced97fa3d6f1146e0cf" member_type: VOTER last_known_addr { host: "127.11.102.3" port: 42987 } }
I20260620 06:06:49.858389 13293 raft_consensus.cc:374] T 03e1ead220b94c319ab7aa71cc62e9c0 P bebfb222b0064148be502c27d76fa23f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 06:06:49.858423 13293 raft_consensus.cc:729] T 03e1ead220b94c319ab7aa71cc62e9c0 P bebfb222b0064148be502c27d76fa23f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bebfb222b0064148be502c27d76fa23f, State: Initialized, Role: FOLLOWER
I20260620 06:06:49.858538 13293 consensus_queue.cc:260] T 03e1ead220b94c319ab7aa71cc62e9c0 P bebfb222b0064148be502c27d76fa23f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "142bdda3511244afba1b84aa91654fc7" member_type: VOTER last_known_addr { host: "127.11.102.1" port: 44353 } } peers { permanent_uuid: "bebfb222b0064148be502c27d76fa23f" member_type: VOTER last_known_addr { host: "127.11.102.2" port: 33501 } } peers { permanent_uuid: "a77868c9753b4ced97fa3d6f1146e0cf" member_type: VOTER last_known_addr { host: "127.11.102.3" port: 42987 } }
I20260620 06:06:49.858798 13293 ts_tablet_manager.cc:1428] T 03e1ead220b94c319ab7aa71cc62e9c0 P bebfb222b0064148be502c27d76fa23f: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260620 06:06:49.859241 13295 tablet_bootstrap.cc:654] T 03e1ead220b94c319ab7aa71cc62e9c0 P a77868c9753b4ced97fa3d6f1146e0cf: Neither blocks nor log segments found. Creating new log.
I20260620 06:06:49.859879 13294 tablet_bootstrap.cc:492] T 03e1ead220b94c319ab7aa71cc62e9c0 P 142bdda3511244afba1b84aa91654fc7: Bootstrap starting.
I20260620 06:06:49.860792 13294 tablet_bootstrap.cc:654] T 03e1ead220b94c319ab7aa71cc62e9c0 P 142bdda3511244afba1b84aa91654fc7: Neither blocks nor log segments found. Creating new log.
I20260620 06:06:49.861248 13316 ts_tablet_manager.cc:1918] T 4be434645ce240faaedc7ad1494249a8 P 142bdda3511244afba1b84aa91654fc7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 06:06:49.861356 13316 log.cc:1196] T 4be434645ce240faaedc7ad1494249a8 P 142bdda3511244afba1b84aa91654fc7: Deleting WAL directory at /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781935603120653-11672-0/minicluster-data/ts-0/wal/wals/4be434645ce240faaedc7ad1494249a8
I20260620 06:06:49.862358 13295 tablet_bootstrap.cc:492] T 03e1ead220b94c319ab7aa71cc62e9c0 P a77868c9753b4ced97fa3d6f1146e0cf: No bootstrap required, opened a new log
I20260620 06:06:49.862440 13295 ts_tablet_manager.cc:1397] T 03e1ead220b94c319ab7aa71cc62e9c0 P a77868c9753b4ced97fa3d6f1146e0cf: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20260620 06:06:49.862921 13295 raft_consensus.cc:348] T 03e1ead220b94c319ab7aa71cc62e9c0 P a77868c9753b4ced97fa3d6f1146e0cf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "142bdda3511244afba1b84aa91654fc7" member_type: VOTER last_known_addr { host: "127.11.102.1" port: 44353 } } peers { permanent_uuid: "bebfb222b0064148be502c27d76fa23f" member_type: VOTER last_known_addr { host: "127.11.102.2" port: 33501 } } peers { permanent_uuid: "a77868c9753b4ced97fa3d6f1146e0cf" member_type: VOTER last_known_addr { host: "127.11.102.3" port: 42987 } }
I20260620 06:06:49.863045 13295 raft_consensus.cc:374] T 03e1ead220b94c319ab7aa71cc62e9c0 P a77868c9753b4ced97fa3d6f1146e0cf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 06:06:49.863085 13295 raft_consensus.cc:729] T 03e1ead220b94c319ab7aa71cc62e9c0 P a77868c9753b4ced97fa3d6f1146e0cf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a77868c9753b4ced97fa3d6f1146e0cf, State: Initialized, Role: FOLLOWER
I20260620 06:06:49.863196 13316 ts_tablet_manager.cc:1939] T 4be434645ce240faaedc7ad1494249a8 P 142bdda3511244afba1b84aa91654fc7: Deleting consensus metadata
I20260620 06:06:49.863219 13295 consensus_queue.cc:260] T 03e1ead220b94c319ab7aa71cc62e9c0 P a77868c9753b4ced97fa3d6f1146e0cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "142bdda3511244afba1b84aa91654fc7" member_type: VOTER last_known_addr { host: "127.11.102.1" port: 44353 } } peers { permanent_uuid: "bebfb222b0064148be502c27d76fa23f" member_type: VOTER last_known_addr { host: "127.11.102.2" port: 33501 } } peers { permanent_uuid: "a77868c9753b4ced97fa3d6f1146e0cf" member_type: VOTER last_known_addr { host: "127.11.102.3" port: 42987 } }
I20260620 06:06:49.863535 13295 ts_tablet_manager.cc:1428] T 03e1ead220b94c319ab7aa71cc62e9c0 P a77868c9753b4ced97fa3d6f1146e0cf: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260620 06:06:49.865165 12774 catalog_manager.cc:4841] TS 142bdda3511244afba1b84aa91654fc7 (127.11.102.1:44353): tablet 4be434645ce240faaedc7ad1494249a8 (table test-table [id=7ae4807bc2094ae2a78a4a649f1c9d0f]) successfully deleted
I20260620 06:06:49.864293 13294 tablet_bootstrap.cc:492] T 03e1ead220b94c319ab7aa71cc62e9c0 P 142bdda3511244afba1b84aa91654fc7: No bootstrap required, opened a new log
I20260620 06:06:49.865401 13294 ts_tablet_manager.cc:1397] T 03e1ead220b94c319ab7aa71cc62e9c0 P 142bdda3511244afba1b84aa91654fc7: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20260620 06:06:49.865847 13294 raft_consensus.cc:348] T 03e1ead220b94c319ab7aa71cc62e9c0 P 142bdda3511244afba1b84aa91654fc7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "142bdda3511244afba1b84aa91654fc7" member_type: VOTER last_known_addr { host: "127.11.102.1" port: 44353 } } peers { permanent_uuid: "bebfb222b0064148be502c27d76fa23f" member_type: VOTER last_known_addr { host: "127.11.102.2" port: 33501 } } peers { permanent_uuid: "a77868c9753b4ced97fa3d6f1146e0cf" member_type: VOTER last_known_addr { host: "127.11.102.3" port: 42987 } }
I20260620 06:06:49.865960 13294 raft_consensus.cc:374] T 03e1ead220b94c319ab7aa71cc62e9c0 P 142bdda3511244afba1b84aa91654fc7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 06:06:49.865999 13294 raft_consensus.cc:729] T 03e1ead220b94c319ab7aa71cc62e9c0 P 142bdda3511244afba1b84aa91654fc7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 142bdda3511244afba1b84aa91654fc7, State: Initialized, Role: FOLLOWER
I20260620 06:06:49.866137 13294 consensus_queue.cc:260] T 03e1ead220b94c319ab7aa71cc62e9c0 P 142bdda3511244afba1b84aa91654fc7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "142bdda3511244afba1b84aa91654fc7" member_type: VOTER last_known_addr { host: "127.11.102.1" port: 44353 } } peers { permanent_uuid: "bebfb222b0064148be502c27d76fa23f" member_type: VOTER last_known_addr { host: "127.11.102.2" port: 33501 } } peers { permanent_uuid: "a77868c9753b4ced97fa3d6f1146e0cf" member_type: VOTER last_known_addr { host: "127.11.102.3" port: 42987 } }
I20260620 06:06:49.866398 13294 ts_tablet_manager.cc:1428] T 03e1ead220b94c319ab7aa71cc62e9c0 P 142bdda3511244afba1b84aa91654fc7: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260620 06:06:49.872934 13299 raft_consensus.cc:482] T 03e1ead220b94c319ab7aa71cc62e9c0 P 142bdda3511244afba1b84aa91654fc7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 06:06:49.873039 13299 raft_consensus.cc:504] T 03e1ead220b94c319ab7aa71cc62e9c0 P 142bdda3511244afba1b84aa91654fc7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "142bdda3511244afba1b84aa91654fc7" member_type: VOTER last_known_addr { host: "127.11.102.1" port: 44353 } } peers { permanent_uuid: "bebfb222b0064148be502c27d76fa23f" member_type: VOTER last_known_addr { host: "127.11.102.2" port: 33501 } } peers { permanent_uuid: "a77868c9753b4ced97fa3d6f1146e0cf" member_type: VOTER last_known_addr { host: "127.11.102.3" port: 42987 } }
I20260620 06:06:49.874158 13299 leader_election.cc:288] T 03e1ead220b94c319ab7aa71cc62e9c0 P 142bdda3511244afba1b84aa91654fc7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bebfb222b0064148be502c27d76fa23f (127.11.102.2:33501), a77868c9753b4ced97fa3d6f1146e0cf (127.11.102.3:42987)
I20260620 06:06:49.878170 13291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:06:49.874439 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.102.1:41339 (local address 127.11.102.2:33501)
0620 06:06:49.874557 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0620 06:06:49.874562 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 06:06:49.874863 (+ 301us) server_negotiation.cc:407] Connection header received
0620 06:06:49.875052 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:06:49.875057 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:06:49.875109 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:06:49.875186 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 06:06:49.875576 (+ 390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:06:49.876217 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:06:49.877239 (+ 1022us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:06:49.877625 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:06:49.877754 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:06:49.877804 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0620 06:06:49.877991 (+ 187us) server_negotiation.cc:299] Negotiation successful
0620 06:06:49.878055 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260620 06:06:49.878185 13322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:06:49.874386 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.102.2:33501 (local address 127.11.102.1:41339)
0620 06:06:49.874772 (+ 386us) negotiation.cc:107] Waiting for socket to connect
0620 06:06:49.874788 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 06:06:49.874928 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:06:49.875302 (+ 374us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:06:49.875305 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:06:49.875322 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 06:06:49.875449 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:06:49.875459 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:06:49.876333 (+ 874us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:06:49.876337 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:06:49.877122 (+ 785us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:06:49.877130 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:06:49.877765 (+ 635us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:06:49.877768 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:06:49.877864 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:06:49.877892 (+ 28us) client_negotiation.cc:769] Sending connection context
0620 06:06:49.877978 (+ 86us) client_negotiation.cc:240] Negotiation successful
0620 06:06:49.878045 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":320,"thread_start_us":96,"threads_started":1}
I20260620 06:06:49.878533 13084 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03e1ead220b94c319ab7aa71cc62e9c0" candidate_uuid: "142bdda3511244afba1b84aa91654fc7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bebfb222b0064148be502c27d76fa23f" is_pre_election: true
I20260620 06:06:49.878652 13084 raft_consensus.cc:2454] T 03e1ead220b94c319ab7aa71cc62e9c0 P bebfb222b0064148be502c27d76fa23f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 142bdda3511244afba1b84aa91654fc7 in term 0.
I20260620 06:06:49.879014 12877 leader_election.cc:302] T 03e1ead220b94c319ab7aa71cc62e9c0 P 142bdda3511244afba1b84aa91654fc7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 142bdda3511244afba1b84aa91654fc7, bebfb222b0064148be502c27d76fa23f; no voters:
I20260620 06:06:49.879287 13323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:06:49.874767 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.102.3:42987 (local address 127.11.102.1:45033)
0620 06:06:49.875176 (+ 409us) negotiation.cc:107] Waiting for socket to connect
0620 06:06:49.875188 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 06:06:49.875256 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:06:49.875512 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:06:49.875515 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:06:49.875532 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 06:06:49.875687 (+ 155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:06:49.875694 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:06:49.876691 (+ 997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:06:49.876694 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:06:49.878041 (+ 1347us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:06:49.878051 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:06:49.878872 (+ 821us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:06:49.878878 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:06:49.878974 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:06:49.878997 (+ 23us) client_negotiation.cc:769] Sending connection context
0620 06:06:49.879078 (+ 81us) client_negotiation.cc:240] Negotiation successful
0620 06:06:49.879141 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":351,"thread_start_us":86,"threads_started":1}
I20260620 06:06:49.879293 13292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:06:49.875138 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.102.1:45033 (local address 127.11.102.3:42987)
0620 06:06:49.875245 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0620 06:06:49.875249 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 06:06:49.875263 (+ 14us) server_negotiation.cc:407] Connection header received
0620 06:06:49.875367 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:06:49.875370 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:06:49.875415 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:06:49.875490 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 06:06:49.875813 (+ 323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:06:49.876556 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:06:49.878172 (+ 1616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:06:49.878773 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:06:49.878891 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:06:49.878936 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0620 06:06:49.879103 (+ 167us) server_negotiation.cc:299] Negotiation successful
0620 06:06:49.879171 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260620 06:06:49.879432 13299 raft_consensus.cc:2781] T 03e1ead220b94c319ab7aa71cc62e9c0 P 142bdda3511244afba1b84aa91654fc7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 06:06:49.879487 13299 raft_consensus.cc:482] T 03e1ead220b94c319ab7aa71cc62e9c0 P 142bdda3511244afba1b84aa91654fc7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 06:06:49.879524 13299 raft_consensus.cc:3037] T 03e1ead220b94c319ab7aa71cc62e9c0 P 142bdda3511244afba1b84aa91654fc7 [term 0 FOLLOWER]: Advancing to term 1
I20260620 06:06:49.880002 13224 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03e1ead220b94c319ab7aa71cc62e9c0" candidate_uuid: "142bdda3511244afba1b84aa91654fc7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a77868c9753b4ced97fa3d6f1146e0cf" is_pre_election: true
I20260620 06:06:49.880259 13224 raft_consensus.cc:2454] T 03e1ead220b94c319ab7aa71cc62e9c0 P a77868c9753b4ced97fa3d6f1146e0cf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 142bdda3511244afba1b84aa91654fc7 in term 0.
I20260620 06:06:49.880672 13299 raft_consensus.cc:504] T 03e1ead220b94c319ab7aa71cc62e9c0 P 142bdda3511244afba1b84aa91654fc7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "142bdda3511244afba1b84aa91654fc7" member_type: VOTER last_known_addr { host: "127.11.102.1" port: 44353 } } peers { permanent_uuid: "bebfb222b0064148be502c27d76fa23f" member_type: VOTER last_known_addr { host: "127.11.102.2" port: 33501 } } peers { permanent_uuid: "a77868c9753b4ced97fa3d6f1146e0cf" member_type: VOTER last_known_addr { host: "127.11.102.3" port: 42987 } }
I20260620 06:06:49.880995 13299 leader_election.cc:288] T 03e1ead220b94c319ab7aa71cc62e9c0 P 142bdda3511244afba1b84aa91654fc7 [CANDIDATE]: Term 1 election: Requested vote from peers bebfb222b0064148be502c27d76fa23f (127.11.102.2:33501), a77868c9753b4ced97fa3d6f1146e0cf (127.11.102.3:42987)
I20260620 06:06:49.881255 13084 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03e1ead220b94c319ab7aa71cc62e9c0" candidate_uuid: "142bdda3511244afba1b84aa91654fc7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bebfb222b0064148be502c27d76fa23f"
I20260620 06:06:49.881323 13224 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03e1ead220b94c319ab7aa71cc62e9c0" candidate_uuid: "142bdda3511244afba1b84aa91654fc7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a77868c9753b4ced97fa3d6f1146e0cf"
I20260620 06:06:49.881359 13084 raft_consensus.cc:3037] T 03e1ead220b94c319ab7aa71cc62e9c0 P bebfb222b0064148be502c27d76fa23f [term 0 FOLLOWER]: Advancing to term 1
I20260620 06:06:49.881417 13224 raft_consensus.cc:3037] T 03e1ead220b94c319ab7aa71cc62e9c0 P a77868c9753b4ced97fa3d6f1146e0cf [term 0 FOLLOWER]: Advancing to term 1
I20260620 06:06:49.882535 13224 raft_consensus.cc:2454] T 03e1ead220b94c319ab7aa71cc62e9c0 P a77868c9753b4ced97fa3d6f1146e0cf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 142bdda3511244afba1b84aa91654fc7 in term 1.
I20260620 06:06:49.882535 13084 raft_consensus.cc:2454] T 03e1ead220b94c319ab7aa71cc62e9c0 P bebfb222b0064148be502c27d76fa23f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 142bdda3511244afba1b84aa91654fc7 in term 1.
I20260620 06:06:49.882786 12877 leader_election.cc:302] T 03e1ead220b94c319ab7aa71cc62e9c0 P 142bdda3511244afba1b84aa91654fc7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 142bdda3511244afba1b84aa91654fc7, a77868c9753b4ced97fa3d6f1146e0cf; no voters:
I20260620 06:06:49.883000 13299 raft_consensus.cc:2781] T 03e1ead220b94c319ab7aa71cc62e9c0 P 142bdda3511244afba1b84aa91654fc7 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 06:06:49.883209 13299 raft_consensus.cc:686] T 03e1ead220b94c319ab7aa71cc62e9c0 P 142bdda3511244afba1b84aa91654fc7 [term 1 LEADER]: Becoming Leader. State: Replica: 142bdda3511244afba1b84aa91654fc7, State: Running, Role: LEADER
I20260620 06:06:49.883466 13299 consensus_queue.cc:237] T 03e1ead220b94c319ab7aa71cc62e9c0 P 142bdda3511244afba1b84aa91654fc7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "142bdda3511244afba1b84aa91654fc7" member_type: VOTER last_known_addr { host: "127.11.102.1" port: 44353 } } peers { permanent_uuid: "bebfb222b0064148be502c27d76fa23f" member_type: VOTER last_known_addr { host: "127.11.102.2" port: 33501 } } peers { permanent_uuid: "a77868c9753b4ced97fa3d6f1146e0cf" member_type: VOTER last_known_addr { host: "127.11.102.3" port: 42987 } }
I20260620 06:06:49.885727 12786 catalog_manager.cc:5495] T 03e1ead220b94c319ab7aa71cc62e9c0 P 142bdda3511244afba1b84aa91654fc7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 142bdda3511244afba1b84aa91654fc7 (127.11.102.1). New cstate: current_term: 1 leader_uuid: "142bdda3511244afba1b84aa91654fc7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "142bdda3511244afba1b84aa91654fc7" member_type: VOTER last_known_addr { host: "127.11.102.1" port: 44353 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bebfb222b0064148be502c27d76fa23f" member_type: VOTER last_known_addr { host: "127.11.102.2" port: 33501 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a77868c9753b4ced97fa3d6f1146e0cf" member_type: VOTER last_known_addr { host: "127.11.102.3" port: 42987 } health_report { overall_health: UNKNOWN } } }
W20260620 06:06:49.887880 13271 tablet.cc:2369] T 03e1ead220b94c319ab7aa71cc62e9c0 P a77868c9753b4ced97fa3d6f1146e0cf: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 06:06:49.910369 13131 tablet.cc:2369] T 03e1ead220b94c319ab7aa71cc62e9c0 P bebfb222b0064148be502c27d76fa23f: Can't schedule compaction. Clean time has not been advanced past its initial value.
Jun 20 06:06:49 dist-test-slave-9hm5 krb5kdc[12742](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935608, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.102.62@KRBTEST.COM for kudu/127.11.102.62@KRBTEST.COM
I20260620 06:06:49.918216 13331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:06:49.909349 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.102.62:39945 (local address 127.0.0.1:55914)
0620 06:06:49.909639 (+ 290us) negotiation.cc:107] Waiting for socket to connect
0620 06:06:49.909656 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 06:06:49.909768 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:06:49.910033 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:06:49.910035 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:06:49.910563 (+ 528us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:06:49.910681 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:06:49.910688 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:06:49.911592 (+ 904us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:06:49.911598 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:06:49.912202 (+ 604us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:06:49.912210 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:06:49.912643 (+ 433us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:06:49.912647 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:06:49.912725 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:06:49.913365 (+ 640us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:06:49.913377 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:06:49.914851 (+ 1474us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:06:49.916762 (+ 1911us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:06:49.916770 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:06:49.916774 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:06:49.916999 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:06:49.917282 (+ 283us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:06:49.917285 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:06:49.917286 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:06:49.917367 (+ 81us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:06:49.917766 (+ 399us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:06:49.917772 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:06:49.917876 (+ 104us) client_negotiation.cc:769] Sending connection context
0620 06:06:49.917969 (+ 93us) client_negotiation.cc:240] Negotiation successful
0620 06:06:49.918079 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":117,"threads_started":1}
I20260620 06:06:49.918339 12836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:06:49.909433 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55914 (local address 127.11.102.62:39945)
0620 06:06:49.909568 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0620 06:06:49.909573 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 06:06:49.909741 (+ 168us) server_negotiation.cc:407] Connection header received
0620 06:06:49.909857 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:06:49.909860 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:06:49.909907 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:06:49.909991 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:06:49.910806 (+ 815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:06:49.911462 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:06:49.912323 (+ 861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:06:49.912536 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:06:49.912644 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:06:49.915039 (+ 2395us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:06:49.915044 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:06:49.915047 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:06:49.915070 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:06:49.916648 (+ 1578us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:06:49.917096 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:06:49.917099 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:06:49.917101 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:06:49.917185 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:06:49.917452 (+ 267us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:06:49.917455 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:06:49.917457 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:06:49.917661 (+ 204us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:06:49.917757 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0620 06:06:49.918041 (+ 284us) server_negotiation.cc:299] Negotiation successful
0620 06:06:49.918152 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260620 06:06:49.922926 12786 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.11.102.62@KRBTEST.COM'} at 127.0.0.1:55914:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260620 06:06:49.923765 12786 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 06:06:49.928511 12924 tablet_service.cc:1467] Processing CreateTablet for tablet 7d6f9ebda2884747a21092dceec286b4 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=54ef38477de947f9b65d557b6f28bf13]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 06:06:49.928490 13064 tablet_service.cc:1467] Processing CreateTablet for tablet 7d6f9ebda2884747a21092dceec286b4 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=54ef38477de947f9b65d557b6f28bf13]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 06:06:49.928874 13064 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7d6f9ebda2884747a21092dceec286b4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 06:06:49.928874 12924 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7d6f9ebda2884747a21092dceec286b4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 06:06:49.931730 13294 tablet_bootstrap.cc:492] T 7d6f9ebda2884747a21092dceec286b4 P 142bdda3511244afba1b84aa91654fc7: Bootstrap starting.
I20260620 06:06:49.932476 13294 tablet_bootstrap.cc:654] T 7d6f9ebda2884747a21092dceec286b4 P 142bdda3511244afba1b84aa91654fc7: Neither blocks nor log segments found. Creating new log.
I20260620 06:06:49.933688 13294 tablet_bootstrap.cc:492] T 7d6f9ebda2884747a21092dceec286b4 P 142bdda3511244afba1b84aa91654fc7: No bootstrap required, opened a new log
I20260620 06:06:49.933754 13294 ts_tablet_manager.cc:1397] T 7d6f9ebda2884747a21092dceec286b4 P 142bdda3511244afba1b84aa91654fc7: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260620 06:06:49.933629 13204 tablet_service.cc:1467] Processing CreateTablet for tablet 7d6f9ebda2884747a21092dceec286b4 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=54ef38477de947f9b65d557b6f28bf13]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 06:06:49.933991 13204 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7d6f9ebda2884747a21092dceec286b4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 06:06:49.934196 13294 raft_consensus.cc:348] T 7d6f9ebda2884747a21092dceec286b4 P 142bdda3511244afba1b84aa91654fc7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "142bdda3511244afba1b84aa91654fc7" member_type: VOTER last_known_addr { host: "127.11.102.1" port: 44353 } } peers { permanent_uuid: "bebfb222b0064148be502c27d76fa23f" member_type: VOTER last_known_addr { host: "127.11.102.2" port: 33501 } } peers { permanent_uuid: "a77868c9753b4ced97fa3d6f1146e0cf" member_type: VOTER last_known_addr { host: "127.11.102.3" port: 42987 } }
I20260620 06:06:49.934329 13294 raft_consensus.cc:374] T 7d6f9ebda2884747a21092dceec286b4 P 142bdda3511244afba1b84aa91654fc7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 06:06:49.934367 13294 raft_consensus.cc:729] T 7d6f9ebda2884747a21092dceec286b4 P 142bdda3511244afba1b84aa91654fc7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 142bdda3511244afba1b84aa91654fc7, State: Initialized, Role: FOLLOWER
I20260620 06:06:49.934484 13294 consensus_queue.cc:260] T 7d6f9ebda2884747a21092dceec286b4 P 142bdda3511244afba1b84aa91654fc7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "142bdda3511244afba1b84aa91654fc7" member_type: VOTER last_known_addr { host: "127.11.102.1" port: 44353 } } peers { permanent_uuid: "bebfb222b0064148be502c27d76fa23f" member_type: VOTER last_known_addr { host: "127.11.102.2" port: 33501 } } peers { permanent_uuid: "a77868c9753b4ced97fa3d6f1146e0cf" member_type: VOTER last_known_addr { host: "127.11.102.3" port: 42987 } }
I20260620 06:06:49.934708 13294 ts_tablet_manager.cc:1428] T 7d6f9ebda2884747a21092dceec286b4 P 142bdda3511244afba1b84aa91654fc7: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260620 06:06:49.935241 13299 tablet_replica.cc:439] T 7d6f9ebda2884747a21092dceec286b4 P 142bdda3511244afba1b84aa91654fc7: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "142bdda3511244afba1b84aa91654fc7" member_type: VOTER last_known_addr { host: "127.11.102.1" port: 44353 } } peers { permanent_uuid: "bebfb222b0064148be502c27d76fa23f" member_type: VOTER last_known_addr { host: "127.11.102.2" port: 33501 } } peers { permanent_uuid: "a77868c9753b4ced97fa3d6f1146e0cf" member_type: VOTER last_known_addr { host: "127.11.102.3" port: 42987 } } }
I20260620 06:06:49.935423 13299 tablet_replica.cc:442] T 7d6f9ebda2884747a21092dceec286b4 P 142bdda3511244afba1b84aa91654fc7: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 06:06:49.935622 13293 tablet_bootstrap.cc:492] T 7d6f9ebda2884747a21092dceec286b4 P bebfb222b0064148be502c27d76fa23f: Bootstrap starting.
I20260620 06:06:49.936580 13293 tablet_bootstrap.cc:654] T 7d6f9ebda2884747a21092dceec286b4 P bebfb222b0064148be502c27d76fa23f: Neither blocks nor log segments found. Creating new log.
I20260620 06:06:49.936723 13295 tablet_bootstrap.cc:492] T 7d6f9ebda2884747a21092dceec286b4 P a77868c9753b4ced97fa3d6f1146e0cf: Bootstrap starting.
I20260620 06:06:49.937517 13295 tablet_bootstrap.cc:654] T 7d6f9ebda2884747a21092dceec286b4 P a77868c9753b4ced97fa3d6f1146e0cf: Neither blocks nor log segments found. Creating new log.
I20260620 06:06:49.937662 13293 tablet_bootstrap.cc:492] T 7d6f9ebda2884747a21092dceec286b4 P bebfb222b0064148be502c27d76fa23f: No bootstrap required, opened a new log
I20260620 06:06:49.937718 13293 ts_tablet_manager.cc:1397] T 7d6f9ebda2884747a21092dceec286b4 P bebfb222b0064148be502c27d76fa23f: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260620 06:06:49.938089 13293 raft_consensus.cc:348] T 7d6f9ebda2884747a21092dceec286b4 P bebfb222b0064148be502c27d76fa23f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "142bdda3511244afba1b84aa91654fc7" member_type: VOTER last_known_addr { host: "127.11.102.1" port: 44353 } } peers { permanent_uuid: "bebfb222b0064148be502c27d76fa23f" member_type: VOTER last_known_addr { host: "127.11.102.2" port: 33501 } } peers { permanent_uuid: "a77868c9753b4ced97fa3d6f1146e0cf" member_type: VOTER last_known_addr { host: "127.11.102.3" port: 42987 } }
I20260620 06:06:49.938210 13293 raft_consensus.cc:374] T 7d6f9ebda2884747a21092dceec286b4 P bebfb222b0064148be502c27d76fa23f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 06:06:49.938249 13293 raft_consensus.cc:729] T 7d6f9ebda2884747a21092dceec286b4 P bebfb222b0064148be502c27d76fa23f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bebfb222b0064148be502c27d76fa23f, State: Initialized, Role: FOLLOWER
I20260620 06:06:49.938396 13293 consensus_queue.cc:260] T 7d6f9ebda2884747a21092dceec286b4 P bebfb222b0064148be502c27d76fa23f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "142bdda3511244afba1b84aa91654fc7" member_type: VOTER last_known_addr { host: "127.11.102.1" port: 44353 } } peers { permanent_uuid: "bebfb222b0064148be502c27d76fa23f" member_type: VOTER last_known_addr { host: "127.11.102.2" port: 33501 } } peers { permanent_uuid: "a77868c9753b4ced97fa3d6f1146e0cf" member_type: VOTER last_known_addr { host: "127.11.102.3" port: 42987 } }
I20260620 06:06:49.938656 13293 ts_tablet_manager.cc:1428] T 7d6f9ebda2884747a21092dceec286b4 P bebfb222b0064148be502c27d76fa23f: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260620 06:06:49.938793 13295 tablet_bootstrap.cc:492] T 7d6f9ebda2884747a21092dceec286b4 P a77868c9753b4ced97fa3d6f1146e0cf: No bootstrap required, opened a new log
I20260620 06:06:49.938882 13295 ts_tablet_manager.cc:1397] T 7d6f9ebda2884747a21092dceec286b4 P a77868c9753b4ced97fa3d6f1146e0cf: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 06:06:49.938901 13301 tablet_replica.cc:439] T 7d6f9ebda2884747a21092dceec286b4 P bebfb222b0064148be502c27d76fa23f: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "142bdda3511244afba1b84aa91654fc7" member_type: VOTER last_known_addr { host: "127.11.102.1" port: 44353 } } peers { permanent_uuid: "bebfb222b0064148be502c27d76fa23f" member_type: VOTER last_known_addr { host: "127.11.102.2" port: 33501 } } peers { permanent_uuid: "a77868c9753b4ced97fa3d6f1146e0cf" member_type: VOTER last_known_addr { host: "127.11.102.3" port: 42987 } } }
I20260620 06:06:49.939039 13301 tablet_replica.cc:442] T 7d6f9ebda2884747a21092dceec286b4 P bebfb222b0064148be502c27d76fa23f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 06:06:49.939278 13295 raft_consensus.cc:348] T 7d6f9ebda2884747a21092dceec286b4 P a77868c9753b4ced97fa3d6f1146e0cf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "142bdda3511244afba1b84aa91654fc7" member_type: VOTER last_known_addr { host: "127.11.102.1" port: 44353 } } peers { permanent_uuid: "bebfb222b0064148be502c27d76fa23f" member_type: VOTER last_known_addr { host: "127.11.102.2" port: 33501 } } peers { permanent_uuid: "a77868c9753b4ced97fa3d6f1146e0cf" member_type: VOTER last_known_addr { host: "127.11.102.3" port: 42987 } }
I20260620 06:06:49.939376 13295 raft_consensus.cc:374] T 7d6f9ebda2884747a21092dceec286b4 P a77868c9753b4ced97fa3d6f1146e0cf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 06:06:49.939414 13295 raft_consensus.cc:729] T 7d6f9ebda2884747a21092dceec286b4 P a77868c9753b4ced97fa3d6f1146e0cf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a77868c9753b4ced97fa3d6f1146e0cf, State: Initialized, Role: FOLLOWER
I20260620 06:06:49.939519 13295 consensus_queue.cc:260] T 7d6f9ebda2884747a21092dceec286b4 P a77868c9753b4ced97fa3d6f1146e0cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "142bdda3511244afba1b84aa91654fc7" member_type: VOTER last_known_addr { host: "127.11.102.1" port: 44353 } } peers { permanent_uuid: "bebfb222b0064148be502c27d76fa23f" member_type: VOTER last_known_addr { host: "127.11.102.2" port: 33501 } } peers { permanent_uuid: "a77868c9753b4ced97fa3d6f1146e0cf" member_type: VOTER last_known_addr { host: "127.11.102.3" port: 42987 } }
I20260620 06:06:49.939788 13295 ts_tablet_manager.cc:1428] T 7d6f9ebda2884747a21092dceec286b4 P a77868c9753b4ced97fa3d6f1146e0cf: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260620 06:06:49.939975 13300 tablet_replica.cc:439] T 7d6f9ebda2884747a21092dceec286b4 P a77868c9753b4ced97fa3d6f1146e0cf: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "142bdda3511244afba1b84aa91654fc7" member_type: VOTER last_known_addr { host: "127.11.102.1" port: 44353 } } peers { permanent_uuid: "bebfb222b0064148be502c27d76fa23f" member_type: VOTER last_known_addr { host: "127.11.102.2" port: 33501 } } peers { permanent_uuid: "a77868c9753b4ced97fa3d6f1146e0cf" member_type: VOTER last_known_addr { host: "127.11.102.3" port: 42987 } } }
I20260620 06:06:49.940091 13300 tablet_replica.cc:442] T 7d6f9ebda2884747a21092dceec286b4 P a77868c9753b4ced97fa3d6f1146e0cf: This TxnStatusTablet replica's current role is: FOLLOWER
W20260620 06:06:49.950662 12991 tablet.cc:2369] T 03e1ead220b94c319ab7aa71cc62e9c0 P 142bdda3511244afba1b84aa91654fc7: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 06:06:50.099668 13299 raft_consensus.cc:482] T 7d6f9ebda2884747a21092dceec286b4 P 142bdda3511244afba1b84aa91654fc7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 06:06:50.099943 13299 raft_consensus.cc:504] T 7d6f9ebda2884747a21092dceec286b4 P 142bdda3511244afba1b84aa91654fc7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "142bdda3511244afba1b84aa91654fc7" member_type: VOTER last_known_addr { host: "127.11.102.1" port: 44353 } } peers { permanent_uuid: "bebfb222b0064148be502c27d76fa23f" member_type: VOTER last_known_addr { host: "127.11.102.2" port: 33501 } } peers { permanent_uuid: "a77868c9753b4ced97fa3d6f1146e0cf" member_type: VOTER last_known_addr { host: "127.11.102.3" port: 42987 } }
I20260620 06:06:50.100420 13299 leader_election.cc:288] T 7d6f9ebda2884747a21092dceec286b4 P 142bdda3511244afba1b84aa91654fc7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bebfb222b0064148be502c27d76fa23f (127.11.102.2:33501), a77868c9753b4ced97fa3d6f1146e0cf (127.11.102.3:42987)
I20260620 06:06:50.100836 13084 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d6f9ebda2884747a21092dceec286b4" candidate_uuid: "142bdda3511244afba1b84aa91654fc7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bebfb222b0064148be502c27d76fa23f" is_pre_election: true
I20260620 06:06:50.100870 13224 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d6f9ebda2884747a21092dceec286b4" candidate_uuid: "142bdda3511244afba1b84aa91654fc7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a77868c9753b4ced97fa3d6f1146e0cf" is_pre_election: true
I20260620 06:06:50.100998 13084 raft_consensus.cc:2454] T 7d6f9ebda2884747a21092dceec286b4 P bebfb222b0064148be502c27d76fa23f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 142bdda3511244afba1b84aa91654fc7 in term 0.
I20260620 06:06:50.101022 13224 raft_consensus.cc:2454] T 7d6f9ebda2884747a21092dceec286b4 P a77868c9753b4ced97fa3d6f1146e0cf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 142bdda3511244afba1b84aa91654fc7 in term 0.
I20260620 06:06:50.101289 12877 leader_election.cc:302] T 7d6f9ebda2884747a21092dceec286b4 P 142bdda3511244afba1b84aa91654fc7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 142bdda3511244afba1b84aa91654fc7, a77868c9753b4ced97fa3d6f1146e0cf; no voters:
I20260620 06:06:50.101468 13299 raft_consensus.cc:2781] T 7d6f9ebda2884747a21092dceec286b4 P 142bdda3511244afba1b84aa91654fc7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 06:06:50.101526 13299 raft_consensus.cc:482] T 7d6f9ebda2884747a21092dceec286b4 P 142bdda3511244afba1b84aa91654fc7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 06:06:50.101552 13299 raft_consensus.cc:3037] T 7d6f9ebda2884747a21092dceec286b4 P 142bdda3511244afba1b84aa91654fc7 [term 0 FOLLOWER]: Advancing to term 1
I20260620 06:06:50.102547 13299 raft_consensus.cc:504] T 7d6f9ebda2884747a21092dceec286b4 P 142bdda3511244afba1b84aa91654fc7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "142bdda3511244afba1b84aa91654fc7" member_type: VOTER last_known_addr { host: "127.11.102.1" port: 44353 } } peers { permanent_uuid: "bebfb222b0064148be502c27d76fa23f" member_type: VOTER last_known_addr { host: "127.11.102.2" port: 33501 } } peers { permanent_uuid: "a77868c9753b4ced97fa3d6f1146e0cf" member_type: VOTER last_known_addr { host: "127.11.102.3" port: 42987 } }
I20260620 06:06:50.102910 13299 leader_election.cc:288] T 7d6f9ebda2884747a21092dceec286b4 P 142bdda3511244afba1b84aa91654fc7 [CANDIDATE]: Term 1 election: Requested vote from peers bebfb222b0064148be502c27d76fa23f (127.11.102.2:33501), a77868c9753b4ced97fa3d6f1146e0cf (127.11.102.3:42987)
I20260620 06:06:50.103253 13084 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d6f9ebda2884747a21092dceec286b4" candidate_uuid: "142bdda3511244afba1b84aa91654fc7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bebfb222b0064148be502c27d76fa23f"
I20260620 06:06:50.103330 13224 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d6f9ebda2884747a21092dceec286b4" candidate_uuid: "142bdda3511244afba1b84aa91654fc7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a77868c9753b4ced97fa3d6f1146e0cf"
I20260620 06:06:50.103390 13084 raft_consensus.cc:3037] T 7d6f9ebda2884747a21092dceec286b4 P bebfb222b0064148be502c27d76fa23f [term 0 FOLLOWER]: Advancing to term 1
I20260620 06:06:50.103456 13224 raft_consensus.cc:3037] T 7d6f9ebda2884747a21092dceec286b4 P a77868c9753b4ced97fa3d6f1146e0cf [term 0 FOLLOWER]: Advancing to term 1
I20260620 06:06:50.104354 13084 raft_consensus.cc:2454] T 7d6f9ebda2884747a21092dceec286b4 P bebfb222b0064148be502c27d76fa23f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 142bdda3511244afba1b84aa91654fc7 in term 1.
I20260620 06:06:50.104507 13224 raft_consensus.cc:2454] T 7d6f9ebda2884747a21092dceec286b4 P a77868c9753b4ced97fa3d6f1146e0cf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 142bdda3511244afba1b84aa91654fc7 in term 1.
I20260620 06:06:50.104664 12877 leader_election.cc:302] T 7d6f9ebda2884747a21092dceec286b4 P 142bdda3511244afba1b84aa91654fc7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 142bdda3511244afba1b84aa91654fc7, bebfb222b0064148be502c27d76fa23f; no voters:
I20260620 06:06:50.104853 13299 raft_consensus.cc:2781] T 7d6f9ebda2884747a21092dceec286b4 P 142bdda3511244afba1b84aa91654fc7 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 06:06:50.104965 13299 raft_consensus.cc:686] T 7d6f9ebda2884747a21092dceec286b4 P 142bdda3511244afba1b84aa91654fc7 [term 1 LEADER]: Becoming Leader. State: Replica: 142bdda3511244afba1b84aa91654fc7, State: Running, Role: LEADER
I20260620 06:06:50.105111 13299 consensus_queue.cc:237] T 7d6f9ebda2884747a21092dceec286b4 P 142bdda3511244afba1b84aa91654fc7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "142bdda3511244afba1b84aa91654fc7" member_type: VOTER last_known_addr { host: "127.11.102.1" port: 44353 } } peers { permanent_uuid: "bebfb222b0064148be502c27d76fa23f" member_type: VOTER last_known_addr { host: "127.11.102.2" port: 33501 } } peers { permanent_uuid: "a77868c9753b4ced97fa3d6f1146e0cf" member_type: VOTER last_known_addr { host: "127.11.102.3" port: 42987 } }
I20260620 06:06:50.105845 13324 tablet_replica.cc:439] T 7d6f9ebda2884747a21092dceec286b4 P 142bdda3511244afba1b84aa91654fc7: TxnStatusTablet state changed. Reason: New leader 142bdda3511244afba1b84aa91654fc7. Latest consensus state: current_term: 1 leader_uuid: "142bdda3511244afba1b84aa91654fc7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "142bdda3511244afba1b84aa91654fc7" member_type: VOTER last_known_addr { host: "127.11.102.1" port: 44353 } } peers { permanent_uuid: "bebfb222b0064148be502c27d76fa23f" member_type: VOTER last_known_addr { host: "127.11.102.2" port: 33501 } } peers { permanent_uuid: "a77868c9753b4ced97fa3d6f1146e0cf" member_type: VOTER last_known_addr { host: "127.11.102.3" port: 42987 } } }
I20260620 06:06:50.105973 13324 tablet_replica.cc:442] T 7d6f9ebda2884747a21092dceec286b4 P 142bdda3511244afba1b84aa91654fc7: This TxnStatusTablet replica's current role is: LEADER
I20260620 06:06:50.106267 13336 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260620 06:06:50.106446 13336 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260620 06:06:50.106856 12787 catalog_manager.cc:5495] T 7d6f9ebda2884747a21092dceec286b4 P 142bdda3511244afba1b84aa91654fc7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 142bdda3511244afba1b84aa91654fc7 (127.11.102.1). New cstate: current_term: 1 leader_uuid: "142bdda3511244afba1b84aa91654fc7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "142bdda3511244afba1b84aa91654fc7" member_type: VOTER last_known_addr { host: "127.11.102.1" port: 44353 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bebfb222b0064148be502c27d76fa23f" member_type: VOTER last_known_addr { host: "127.11.102.2" port: 33501 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a77868c9753b4ced97fa3d6f1146e0cf" member_type: VOTER last_known_addr { host: "127.11.102.3" port: 42987 } health_report { overall_health: UNKNOWN } } }
I20260620 06:06:50.134493 13331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:06:50.130335 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.102.1:44353 (local address 127.0.0.1:60892)
0620 06:06:50.130526 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0620 06:06:50.130551 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0620 06:06:50.130637 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:06:50.130959 (+ 322us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:06:50.130961 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:06:50.130967 (+ 6us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 06:06:50.131065 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:06:50.131070 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:06:50.132106 (+ 1036us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:06:50.132110 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:06:50.132720 (+ 610us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:06:50.132728 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:06:50.133303 (+ 575us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:06:50.133306 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:06:50.133377 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:06:50.133420 (+ 43us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 06:06:50.134209 (+ 789us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 06:06:50.134225 (+ 16us) client_negotiation.cc:769] Sending connection context
0620 06:06:50.134303 (+ 78us) client_negotiation.cc:240] Negotiation successful
0620 06:06:50.134378 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":77}
I20260620 06:06:50.134531 13289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:06:50.130470 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60892 (local address 127.11.102.1:44353)
0620 06:06:50.130591 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0620 06:06:50.130596 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 06:06:50.130612 (+ 16us) server_negotiation.cc:407] Connection header received
0620 06:06:50.130775 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:06:50.130778 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:06:50.130842 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:06:50.130946 (+ 104us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 06:06:50.131165 (+ 219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:06:50.131975 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:06:50.132832 (+ 857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:06:50.133168 (+ 336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:06:50.133272 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:06:50.133550 (+ 278us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 06:06:50.134059 (+ 509us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 06:06:50.134170 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0620 06:06:50.134332 (+ 162us) server_negotiation.cc:299] Negotiation successful
0620 06:06:50.134385 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260620 06:06:50.137640 13084 raft_consensus.cc:1261] T 7d6f9ebda2884747a21092dceec286b4 P bebfb222b0064148be502c27d76fa23f [term 1 FOLLOWER]: Refusing update from remote peer 142bdda3511244afba1b84aa91654fc7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 06:06:50.137795 13224 raft_consensus.cc:1261] T 7d6f9ebda2884747a21092dceec286b4 P a77868c9753b4ced97fa3d6f1146e0cf [term 1 FOLLOWER]: Refusing update from remote peer 142bdda3511244afba1b84aa91654fc7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 06:06:50.138152 13299 consensus_queue.cc:1035] T 7d6f9ebda2884747a21092dceec286b4 P 142bdda3511244afba1b84aa91654fc7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bebfb222b0064148be502c27d76fa23f" member_type: VOTER last_known_addr { host: "127.11.102.2" port: 33501 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 06:06:50.138334 13324 consensus_queue.cc:1035] T 7d6f9ebda2884747a21092dceec286b4 P 142bdda3511244afba1b84aa91654fc7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a77868c9753b4ced97fa3d6f1146e0cf" member_type: VOTER last_known_addr { host: "127.11.102.3" port: 42987 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 06:06:50.139714 13301 tablet_replica.cc:439] T 7d6f9ebda2884747a21092dceec286b4 P bebfb222b0064148be502c27d76fa23f: TxnStatusTablet state changed. Reason: New leader 142bdda3511244afba1b84aa91654fc7. Latest consensus state: current_term: 1 leader_uuid: "142bdda3511244afba1b84aa91654fc7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "142bdda3511244afba1b84aa91654fc7" member_type: VOTER last_known_addr { host: "127.11.102.1" port: 44353 } } peers { permanent_uuid: "bebfb222b0064148be502c27d76fa23f" member_type: VOTER last_known_addr { host: "127.11.102.2" port: 33501 } } peers { permanent_uuid: "a77868c9753b4ced97fa3d6f1146e0cf" member_type: VOTER last_known_addr { host: "127.11.102.3" port: 42987 } } }
I20260620 06:06:50.139823 13301 tablet_replica.cc:442] T 7d6f9ebda2884747a21092dceec286b4 P bebfb222b0064148be502c27d76fa23f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 06:06:50.140707 13300 tablet_replica.cc:439] T 7d6f9ebda2884747a21092dceec286b4 P a77868c9753b4ced97fa3d6f1146e0cf: TxnStatusTablet state changed. Reason: New leader 142bdda3511244afba1b84aa91654fc7. Latest consensus state: current_term: 1 leader_uuid: "142bdda3511244afba1b84aa91654fc7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "142bdda3511244afba1b84aa91654fc7" member_type: VOTER last_known_addr { host: "127.11.102.1" port: 44353 } } peers { permanent_uuid: "bebfb222b0064148be502c27d76fa23f" member_type: VOTER last_known_addr { host: "127.11.102.2" port: 33501 } } peers { permanent_uuid: "a77868c9753b4ced97fa3d6f1146e0cf" member_type: VOTER last_known_addr { host: "127.11.102.3" port: 42987 } } }
I20260620 06:06:50.140825 13300 tablet_replica.cc:442] T 7d6f9ebda2884747a21092dceec286b4 P a77868c9753b4ced97fa3d6f1146e0cf: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 06:06:50.142167 13299 tablet_replica.cc:439] T 7d6f9ebda2884747a21092dceec286b4 P 142bdda3511244afba1b84aa91654fc7: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "142bdda3511244afba1b84aa91654fc7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "142bdda3511244afba1b84aa91654fc7" member_type: VOTER last_known_addr { host: "127.11.102.1" port: 44353 } } peers { permanent_uuid: "bebfb222b0064148be502c27d76fa23f" member_type: VOTER last_known_addr { host: "127.11.102.2" port: 33501 } } peers { permanent_uuid: "a77868c9753b4ced97fa3d6f1146e0cf" member_type: VOTER last_known_addr { host: "127.11.102.3" port: 42987 } } }
I20260620 06:06:50.142270 13299 tablet_replica.cc:442] T 7d6f9ebda2884747a21092dceec286b4 P 142bdda3511244afba1b84aa91654fc7: This TxnStatusTablet replica's current role is: LEADER
I20260620 06:06:50.143716 13308 mvcc.cc:205] Tried to move back new op lower bound from 7298808259119587328 to 7298808258992869376. Current Snapshot: MvccSnapshot[applied={T|T < 7298808259119587328}]
I20260620 06:06:50.144326 13299 tablet_replica.cc:439] T 7d6f9ebda2884747a21092dceec286b4 P 142bdda3511244afba1b84aa91654fc7: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "142bdda3511244afba1b84aa91654fc7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "142bdda3511244afba1b84aa91654fc7" member_type: VOTER last_known_addr { host: "127.11.102.1" port: 44353 } } peers { permanent_uuid: "bebfb222b0064148be502c27d76fa23f" member_type: VOTER last_known_addr { host: "127.11.102.2" port: 33501 } } peers { permanent_uuid: "a77868c9753b4ced97fa3d6f1146e0cf" member_type: VOTER last_known_addr { host: "127.11.102.3" port: 42987 } } }
I20260620 06:06:50.144428 13299 tablet_replica.cc:442] T 7d6f9ebda2884747a21092dceec286b4 P 142bdda3511244afba1b84aa91654fc7: This TxnStatusTablet replica's current role is: LEADER
I20260620 06:06:50.144820 13301 tablet_replica.cc:439] T 7d6f9ebda2884747a21092dceec286b4 P bebfb222b0064148be502c27d76fa23f: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "142bdda3511244afba1b84aa91654fc7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "142bdda3511244afba1b84aa91654fc7" member_type: VOTER last_known_addr { host: "127.11.102.1" port: 44353 } } peers { permanent_uuid: "bebfb222b0064148be502c27d76fa23f" member_type: VOTER last_known_addr { host: "127.11.102.2" port: 33501 } } peers { permanent_uuid: "a77868c9753b4ced97fa3d6f1146e0cf" member_type: VOTER last_known_addr { host: "127.11.102.3" port: 42987 } } }
I20260620 06:06:50.144923 13301 tablet_replica.cc:442] T 7d6f9ebda2884747a21092dceec286b4 P bebfb222b0064148be502c27d76fa23f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 06:06:50.144979 13300 tablet_replica.cc:439] T 7d6f9ebda2884747a21092dceec286b4 P a77868c9753b4ced97fa3d6f1146e0cf: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "142bdda3511244afba1b84aa91654fc7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "142bdda3511244afba1b84aa91654fc7" member_type: VOTER last_known_addr { host: "127.11.102.1" port: 44353 } } peers { permanent_uuid: "bebfb222b0064148be502c27d76fa23f" member_type: VOTER last_known_addr { host: "127.11.102.2" port: 33501 } } peers { permanent_uuid: "a77868c9753b4ced97fa3d6f1146e0cf" member_type: VOTER last_known_addr { host: "127.11.102.3" port: 42987 } } }
I20260620 06:06:50.145081 13300 tablet_replica.cc:442] T 7d6f9ebda2884747a21092dceec286b4 P a77868c9753b4ced97fa3d6f1146e0cf: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 06:06:50.152608 13289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:06:50.148992 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60904 (local address 127.11.102.1:44353)
0620 06:06:50.149097 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0620 06:06:50.149101 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 06:06:50.149114 (+ 13us) server_negotiation.cc:407] Connection header received
0620 06:06:50.149220 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:06:50.149223 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:06:50.149266 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:06:50.149346 (+ 80us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 06:06:50.149522 (+ 176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:06:50.150292 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:06:50.151062 (+ 770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:06:50.151844 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:06:50.151938 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:06:50.152201 (+ 263us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 06:06:50.152283 (+ 82us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 06:06:50.152352 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0620 06:06:50.152434 (+ 82us) server_negotiation.cc:299] Negotiation successful
0620 06:06:50.152482 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260620 06:06:50.162883 13289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:06:50.159274 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.102.1:38391 (local address 127.11.102.1:44353)
0620 06:06:50.159394 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0620 06:06:50.159398 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 06:06:50.159576 (+ 178us) server_negotiation.cc:407] Connection header received
0620 06:06:50.159713 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:06:50.159716 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:06:50.159753 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:06:50.159811 (+ 58us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 06:06:50.160071 (+ 260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:06:50.160570 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:06:50.161839 (+ 1269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:06:50.162053 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:06:50.162127 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:06:50.162447 (+ 320us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 06:06:50.162512 (+ 65us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 06:06:50.162567 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0620 06:06:50.162732 (+ 165us) server_negotiation.cc:299] Negotiation successful
0620 06:06:50.162770 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260620 06:06:50.163076 13343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:06:50.159224 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.102.1:44353 (local address 127.11.102.1:38391)
0620 06:06:50.159489 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0620 06:06:50.159503 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 06:06:50.159598 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:06:50.159842 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:06:50.159845 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:06:50.159852 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 06:06:50.159973 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:06:50.159978 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:06:50.160687 (+ 709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:06:50.160691 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:06:50.161688 (+ 997us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:06:50.161707 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:06:50.162168 (+ 461us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:06:50.162172 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:06:50.162262 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:06:50.162309 (+ 47us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 06:06:50.162616 (+ 307us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 06:06:50.162621 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 06:06:50.162688 (+ 67us) client_negotiation.cc:240] Negotiation successful
0620 06:06:50.162768 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":97,"threads_started":1}
I20260620 06:06:50.169626 13224 raft_consensus.cc:1261] T 03e1ead220b94c319ab7aa71cc62e9c0 P a77868c9753b4ced97fa3d6f1146e0cf [term 1 FOLLOWER]: Refusing update from remote peer 142bdda3511244afba1b84aa91654fc7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 06:06:50.169656 13084 raft_consensus.cc:1261] T 03e1ead220b94c319ab7aa71cc62e9c0 P bebfb222b0064148be502c27d76fa23f [term 1 FOLLOWER]: Refusing update from remote peer 142bdda3511244afba1b84aa91654fc7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 06:06:50.170006 13299 consensus_queue.cc:1035] T 03e1ead220b94c319ab7aa71cc62e9c0 P 142bdda3511244afba1b84aa91654fc7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a77868c9753b4ced97fa3d6f1146e0cf" member_type: VOTER last_known_addr { host: "127.11.102.3" port: 42987 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 06:06:50.170122 13324 consensus_queue.cc:1035] T 03e1ead220b94c319ab7aa71cc62e9c0 P 142bdda3511244afba1b84aa91654fc7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bebfb222b0064148be502c27d76fa23f" member_type: VOTER last_known_addr { host: "127.11.102.2" port: 33501 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 06:06:50.177698 13306 mvcc.cc:205] Tried to move back new op lower bound from 7298808259251363840 to 7298808258085896192. Current Snapshot: MvccSnapshot[applied={T|T < 7298808259251363840}]
I20260620 06:06:50.215672 12785 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55912:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 06:06:50.215842 12785 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55912:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 06:06:50.217965 12785 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6e6f3d203fc348ba9e349e81d952fc19: Sending DeleteTablet for 3 replicas of tablet 03e1ead220b94c319ab7aa71cc62e9c0
I20260620 06:06:50.218406 12923 tablet_service.cc:1514] Processing DeleteTablet for tablet 03e1ead220b94c319ab7aa71cc62e9c0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 06:06:50 UTC) from {username='kudu', principal='kudu/127.11.102.62@KRBTEST.COM'} at 127.0.0.1:60878
I20260620 06:06:50.218564 13316 tablet_replica.cc:330] T 03e1ead220b94c319ab7aa71cc62e9c0 P 142bdda3511244afba1b84aa91654fc7: stopping tablet replica
I20260620 06:06:50.218624 13064 tablet_service.cc:1514] Processing DeleteTablet for tablet 03e1ead220b94c319ab7aa71cc62e9c0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 06:06:50 UTC) from {username='kudu', principal='kudu/127.11.102.62@KRBTEST.COM'} at 127.0.0.1:46098
I20260620 06:06:50.218683 13316 raft_consensus.cc:2229] T 03e1ead220b94c319ab7aa71cc62e9c0 P 142bdda3511244afba1b84aa91654fc7 [term 1 LEADER]: Raft consensus shutting down.
I20260620 06:06:50.218741 13204 tablet_service.cc:1514] Processing DeleteTablet for tablet 03e1ead220b94c319ab7aa71cc62e9c0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 06:06:50 UTC) from {username='kudu', principal='kudu/127.11.102.62@KRBTEST.COM'} at 127.0.0.1:56992
I20260620 06:06:50.218811 13318 tablet_replica.cc:330] T 03e1ead220b94c319ab7aa71cc62e9c0 P bebfb222b0064148be502c27d76fa23f: stopping tablet replica
I20260620 06:06:50.218912 13317 tablet_replica.cc:330] T 03e1ead220b94c319ab7aa71cc62e9c0 P a77868c9753b4ced97fa3d6f1146e0cf: stopping tablet replica
I20260620 06:06:50.218912 13318 raft_consensus.cc:2229] T 03e1ead220b94c319ab7aa71cc62e9c0 P bebfb222b0064148be502c27d76fa23f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 06:06:50.218950 13316 raft_consensus.cc:2258] T 03e1ead220b94c319ab7aa71cc62e9c0 P 142bdda3511244afba1b84aa91654fc7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 06:06:50.219008 13318 raft_consensus.cc:2258] T 03e1ead220b94c319ab7aa71cc62e9c0 P bebfb222b0064148be502c27d76fa23f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 06:06:50.219028 13317 raft_consensus.cc:2229] T 03e1ead220b94c319ab7aa71cc62e9c0 P a77868c9753b4ced97fa3d6f1146e0cf [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 06:06:50.219133 13317 raft_consensus.cc:2258] T 03e1ead220b94c319ab7aa71cc62e9c0 P a77868c9753b4ced97fa3d6f1146e0cf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 06:06:50.219727 13316 ts_tablet_manager.cc:1905] T 03e1ead220b94c319ab7aa71cc62e9c0 P 142bdda3511244afba1b84aa91654fc7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 06:06:50.219795 13318 ts_tablet_manager.cc:1905] T 03e1ead220b94c319ab7aa71cc62e9c0 P bebfb222b0064148be502c27d76fa23f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 06:06:50.222208 13316 ts_tablet_manager.cc:1918] T 03e1ead220b94c319ab7aa71cc62e9c0 P 142bdda3511244afba1b84aa91654fc7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 06:06:50.222287 13316 log.cc:1196] T 03e1ead220b94c319ab7aa71cc62e9c0 P 142bdda3511244afba1b84aa91654fc7: Deleting WAL directory at /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781935603120653-11672-0/minicluster-data/ts-0/wal/wals/03e1ead220b94c319ab7aa71cc62e9c0
I20260620 06:06:50.222471 13318 ts_tablet_manager.cc:1918] T 03e1ead220b94c319ab7aa71cc62e9c0 P bebfb222b0064148be502c27d76fa23f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 06:06:50.222553 13318 log.cc:1196] T 03e1ead220b94c319ab7aa71cc62e9c0 P bebfb222b0064148be502c27d76fa23f: Deleting WAL directory at /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781935603120653-11672-0/minicluster-data/ts-1/wal/wals/03e1ead220b94c319ab7aa71cc62e9c0
I20260620 06:06:50.222574 13316 ts_tablet_manager.cc:1939] T 03e1ead220b94c319ab7aa71cc62e9c0 P 142bdda3511244afba1b84aa91654fc7: Deleting consensus metadata
I20260620 06:06:50.222815 13318 ts_tablet_manager.cc:1939] T 03e1ead220b94c319ab7aa71cc62e9c0 P bebfb222b0064148be502c27d76fa23f: Deleting consensus metadata
I20260620 06:06:50.223258 12774 catalog_manager.cc:4841] TS 142bdda3511244afba1b84aa91654fc7 (127.11.102.1:44353): tablet 03e1ead220b94c319ab7aa71cc62e9c0 (table test-table [id=b1000c9283134c26913891dbb0ac09f1]) successfully deleted
I20260620 06:06:50.223419 12773 catalog_manager.cc:4841] TS bebfb222b0064148be502c27d76fa23f (127.11.102.2:33501): tablet 03e1ead220b94c319ab7aa71cc62e9c0 (table test-table [id=b1000c9283134c26913891dbb0ac09f1]) successfully deleted
I20260620 06:06:50.224133 13317 ts_tablet_manager.cc:1905] T 03e1ead220b94c319ab7aa71cc62e9c0 P a77868c9753b4ced97fa3d6f1146e0cf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 06:06:50.226675 13317 ts_tablet_manager.cc:1918] T 03e1ead220b94c319ab7aa71cc62e9c0 P a77868c9753b4ced97fa3d6f1146e0cf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 06:06:50.226756 13317 log.cc:1196] T 03e1ead220b94c319ab7aa71cc62e9c0 P a77868c9753b4ced97fa3d6f1146e0cf: Deleting WAL directory at /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781935603120653-11672-0/minicluster-data/ts-2/wal/wals/03e1ead220b94c319ab7aa71cc62e9c0
I20260620 06:06:50.226997 13317 ts_tablet_manager.cc:1939] T 03e1ead220b94c319ab7aa71cc62e9c0 P a77868c9753b4ced97fa3d6f1146e0cf: Deleting consensus metadata
I20260620 06:06:50.227638 12773 catalog_manager.cc:4841] TS a77868c9753b4ced97fa3d6f1146e0cf (127.11.102.3:42987): tablet 03e1ead220b94c319ab7aa71cc62e9c0 (table test-table [id=b1000c9283134c26913891dbb0ac09f1]) successfully deleted
Jun 20 06:06:50 dist-test-slave-9hm5 krb5kdc[12742](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935609, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.11.102.1@KRBTEST.COM
I20260620 06:06:50.238922 13289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:06:50.227174 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60908 (local address 127.11.102.1:44353)
0620 06:06:50.227278 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0620 06:06:50.227283 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 06:06:50.227857 (+ 574us) server_negotiation.cc:407] Connection header received
0620 06:06:50.228014 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:06:50.228018 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:06:50.228065 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:06:50.228149 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:06:50.229354 (+ 1205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:06:50.230154 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:06:50.230950 (+ 796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:06:50.231286 (+ 336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:06:50.231392 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:06:50.234614 (+ 3222us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:06:50.234623 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:06:50.234641 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:06:50.234676 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:06:50.237150 (+ 2474us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:06:50.237578 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:06:50.237586 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:06:50.237587 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:06:50.237690 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:06:50.237939 (+ 249us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:06:50.237943 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:06:50.237945 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:06:50.238246 (+ 301us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:06:50.238463 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0620 06:06:50.238597 (+ 134us) server_negotiation.cc:299] Negotiation successful
0620 06:06:50.238725 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
W20260620 06:06:50.239470 12984 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:60908
I20260620 06:06:50.249034 12836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:06:50.241298 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55918 (local address 127.11.102.62:39945)
0620 06:06:50.241417 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0620 06:06:50.241423 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 06:06:50.241510 (+ 87us) server_negotiation.cc:407] Connection header received
0620 06:06:50.241607 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:06:50.241610 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:06:50.241669 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:06:50.241758 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:06:50.242336 (+ 578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:06:50.243203 (+ 867us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:06:50.244004 (+ 801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:06:50.244341 (+ 337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:06:50.244450 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:06:50.245686 (+ 1236us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:06:50.245690 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:06:50.245692 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:06:50.245719 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:06:50.247247 (+ 1528us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:06:50.247746 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:06:50.247750 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:06:50.247752 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:06:50.247840 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:06:50.248147 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:06:50.248150 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:06:50.248152 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:06:50.248372 (+ 220us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:06:50.248509 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0620 06:06:50.248754 (+ 245us) server_negotiation.cc:299] Negotiation successful
0620 06:06:50.248872 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
W20260620 06:06:50.249502 12785 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55918
I20260620 06:06:50.250934 11672 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9HMp43/build/debug/bin/kudu with pid 12859
I20260620 06:06:50.257094 11672 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9HMp43/build/debug/bin/kudu with pid 12999
I20260620 06:06:50.262706 11672 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9HMp43/build/debug/bin/kudu with pid 13139
I20260620 06:06:50.269029 11672 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9HMp43/build/debug/bin/kudu with pid 12755
2026-06-20T06:06:50Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (2654 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781935603120653-11672-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 06:06:50 dist-test-slave-9hm5 krb5kdc[13355](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 06:06:50 dist-test-slave-9hm5 krb5kdc[13355](info): set up 2 sockets
Jun 20 06:06:50 dist-test-slave-9hm5 krb5kdc[13355](info): commencing operation
krb5kdc: starting...
W20260620 06:06:51.291302 11672 mini_kdc.cc:121] Time spent starting KDC: real 1.001s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781935603120653-11672-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781935603120653-11672-0/krb5kdc/test-user.keytab.
Jun 20 06:06:51 dist-test-slave-9hm5 krb5kdc[13355](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935611, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T06:06:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T06:06:51Z Disabled control of system clock
WARNING: no policy specified for kudu/127.11.102.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.102.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.102.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781935603120653-11672-0/krb5kdc/kudu_127.11.102.62.keytab.
Entry for principal kudu/127.11.102.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781935603120653-11672-0/krb5kdc/kudu_127.11.102.62.keytab.
I20260620 06:06:51.418376 11672 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9HMp43/build/debug/bin/kudu
/tmp/dist-test-task9HMp43/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781935603120653-11672-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781935603120653-11672-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781935603120653-11672-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781935603120653-11672-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.102.62:39839
--webserver_interface=127.11.102.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781935603120653-11672-0/krb5kdc/kudu_127.11.102.62.keytab
--principal=kudu/127.11.102.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.102.20:33411
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.102.62:39839
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781935603120653-11672-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781935603120653-11672-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781935603120653-11672-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 06:06:51.523438 13368 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:06:51.523749 13368 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:06:51.523823 13368 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:06:51.528333 13368 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 06:06:51.528456 13368 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260620 06:06:51.528607 13368 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:06:51.528698 13368 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:06:51.528743 13368 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 06:06:51.528777 13368 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 06:06:51.534212 13368 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.102.20:33411
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781935603120653-11672-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781935603120653-11672-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.102.62:39839
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781935603120653-11672-0/krb5kdc/kudu_127.11.102.62.keytab
--principal=kudu/127.11.102.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.102.62:39839
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781935603120653-11672-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.102.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13368
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781935603120653-11672-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 05:28:48 UTC on ccycloud.quasar-fixhms.root.comops.site
build id 4457020
I20260620 06:06:51.535224 13368 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:06:51.536237 13368 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 06:06:51.544400 13374 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:06:51.544394 13376 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:06:51.544394 13373 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:06:51.544653 13368 server_base.cc:1034] running on GCE node
I20260620 06:06:51.545212 13368 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:06:51.546271 13368 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:06:51.547456 13368 hybrid_clock.cc:648] HybridClock initialized: now 1781935611547444 us; error 23 us; skew 500 ppm
Jun 20 06:06:51 dist-test-slave-9hm5 krb5kdc[13355](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935611, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.102.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 06:06:51.550065 13368 init.cc:377] Logged in from keytab as kudu/127.11.102.62@KRBTEST.COM (short username kudu)
I20260620 06:06:51.550879 13368 webserver.cc:460] Webserver started at http://127.11.102.62:34925/ using document root <none> and password file <none>
I20260620 06:06:51.551355 13368 fs_manager.cc:359] Metadata directory not provided
I20260620 06:06:51.551410 13368 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:06:51.551584 13368 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 06:06:51.553508 13368 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781935603120653-11672-0/minicluster-data/master-0/data/instance:
uuid: "794f593770bb429487ad0ec805dae6bf"
format_stamp: "Formatted at 2026-06-20 06:06:51 on dist-test-slave-9hm5"
server_key: "e319c854e32dd546e49b0f487416d9b7"
server_key_iv: "3c2533ed66ffe4b9106e0ea7c851fea7"
server_key_version: "encryptionkey@0"
I20260620 06:06:51.553987 13368 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781935603120653-11672-0/minicluster-data/master-0/wal/instance:
uuid: "794f593770bb429487ad0ec805dae6bf"
format_stamp: "Formatted at 2026-06-20 06:06:51 on dist-test-slave-9hm5"
server_key: "e319c854e32dd546e49b0f487416d9b7"
server_key_iv: "3c2533ed66ffe4b9106e0ea7c851fea7"
server_key_version: "encryptionkey@0"
I20260620 06:06:51.557453 13368 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.005s
I20260620 06:06:51.559664 13383 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:06:51.560820 13368 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.002s
I20260620 06:06:51.560915 13368 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781935603120653-11672-0/minicluster-data/master-0/data,/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781935603120653-11672-0/minicluster-data/master-0/wal
uuid: "794f593770bb429487ad0ec805dae6bf"
format_stamp: "Formatted at 2026-06-20 06:06:51 on dist-test-slave-9hm5"
server_key: "e319c854e32dd546e49b0f487416d9b7"
server_key_iv: "3c2533ed66ffe4b9106e0ea7c851fea7"
server_key_version: "encryptionkey@0"
I20260620 06:06:51.560997 13368 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781935603120653-11672-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781935603120653-11672-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781935603120653-11672-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:06:51.589607 13368 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:06:51.592981 13368 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:06:51.593163 13368 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:06:51.600827 13368 rpc_server.cc:307] RPC server started. Bound to: 127.11.102.62:39839
I20260620 06:06:51.600848 13445 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.102.62:39839 every 8 connection(s)
I20260620 06:06:51.601753 13368 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781935603120653-11672-0/minicluster-data/master-0/data/info.pb
I20260620 06:06:51.604369 11672 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9HMp43/build/debug/bin/kudu as pid 13368
I20260620 06:06:51.604461 11672 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781935603120653-11672-0/minicluster-data/master-0/wal/instance
I20260620 06:06:51.604558 13446 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 06:06:51.604789 11672 external_mini_cluster.cc:1428] Setting key c933e27ec907ff6cceb125625e3cf39d
I20260620 06:06:51.610805 13446 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 794f593770bb429487ad0ec805dae6bf: Bootstrap starting.
I20260620 06:06:51.613178 13446 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 794f593770bb429487ad0ec805dae6bf: Neither blocks nor log segments found. Creating new log.
Jun 20 06:06:51 dist-test-slave-9hm5 krb5kdc[13355](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935611, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.102.62@KRBTEST.COM
I20260620 06:06:51.613947 13446 log.cc:824] T 00000000000000000000000000000000 P 794f593770bb429487ad0ec805dae6bf: Log is configured to *not* fsync() on all Append() calls
I20260620 06:06:51.616564 13446 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 794f593770bb429487ad0ec805dae6bf: No bootstrap required, opened a new log
I20260620 06:06:51.619580 13449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:06:51.606136 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57808 (local address 127.11.102.62:39839)
0620 06:06:51.606555 (+ 419us) server_negotiation.cc:206] Beginning negotiation
0620 06:06:51.606569 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 06:06:51.606603 (+ 34us) server_negotiation.cc:407] Connection header received
0620 06:06:51.607202 (+ 599us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:06:51.607218 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:06:51.607533 (+ 315us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:06:51.607918 (+ 385us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:06:51.608710 (+ 792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:06:51.609695 (+ 985us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:06:51.610560 (+ 865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:06:51.611083 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:06:51.611249 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:06:51.613908 (+ 2659us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:06:51.613914 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:06:51.613927 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:06:51.613953 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:06:51.616598 (+ 2645us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:06:51.617066 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:06:51.617072 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:06:51.617078 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:06:51.617221 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:06:51.617475 (+ 254us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:06:51.617478 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:06:51.617480 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:06:51.617955 (+ 475us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:06:51.618159 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0620 06:06:51.618349 (+ 190us) server_negotiation.cc:299] Negotiation successful
0620 06:06:51.618647 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":286,"thread_start_us":136,"threads_started":1}
I20260620 06:06:51.620368 13446 raft_consensus.cc:348] T 00000000000000000000000000000000 P 794f593770bb429487ad0ec805dae6bf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "794f593770bb429487ad0ec805dae6bf" member_type: VOTER last_known_addr { host: "127.11.102.62" port: 39839 } }
I20260620 06:06:51.620606 13446 raft_consensus.cc:374] T 00000000000000000000000000000000 P 794f593770bb429487ad0ec805dae6bf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 06:06:51.620666 13446 raft_consensus.cc:729] T 00000000000000000000000000000000 P 794f593770bb429487ad0ec805dae6bf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 794f593770bb429487ad0ec805dae6bf, State: Initialized, Role: FOLLOWER
I20260620 06:06:51.621059 13446 consensus_queue.cc:260] T 00000000000000000000000000000000 P 794f593770bb429487ad0ec805dae6bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "794f593770bb429487ad0ec805dae6bf" member_type: VOTER last_known_addr { host: "127.11.102.62" port: 39839 } }
I20260620 06:06:51.621217 13446 raft_consensus.cc:388] T 00000000000000000000000000000000 P 794f593770bb429487ad0ec805dae6bf [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 06:06:51.621294 13446 raft_consensus.cc:482] T 00000000000000000000000000000000 P 794f593770bb429487ad0ec805dae6bf [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 06:06:51.621387 13446 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 794f593770bb429487ad0ec805dae6bf [term 0 FOLLOWER]: Advancing to term 1
I20260620 06:06:51.622346 13446 raft_consensus.cc:504] T 00000000000000000000000000000000 P 794f593770bb429487ad0ec805dae6bf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "794f593770bb429487ad0ec805dae6bf" member_type: VOTER last_known_addr { host: "127.11.102.62" port: 39839 } }
I20260620 06:06:51.622680 13446 leader_election.cc:302] T 00000000000000000000000000000000 P 794f593770bb429487ad0ec805dae6bf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 794f593770bb429487ad0ec805dae6bf; no voters:
I20260620 06:06:51.622984 13446 leader_election.cc:288] T 00000000000000000000000000000000 P 794f593770bb429487ad0ec805dae6bf [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 06:06:51.623095 13451 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 794f593770bb429487ad0ec805dae6bf [term 1 FOLLOWER]: Leader election won for term 1
I20260620 06:06:51.623337 13451 raft_consensus.cc:686] T 00000000000000000000000000000000 P 794f593770bb429487ad0ec805dae6bf [term 1 LEADER]: Becoming Leader. State: Replica: 794f593770bb429487ad0ec805dae6bf, State: Running, Role: LEADER
I20260620 06:06:51.623659 13451 consensus_queue.cc:237] T 00000000000000000000000000000000 P 794f593770bb429487ad0ec805dae6bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "794f593770bb429487ad0ec805dae6bf" member_type: VOTER last_known_addr { host: "127.11.102.62" port: 39839 } }
I20260620 06:06:51.624241 13446 sys_catalog.cc:564] T 00000000000000000000000000000000 P 794f593770bb429487ad0ec805dae6bf [sys.catalog]: configured and running, proceeding with master startup.
I20260620 06:06:51.625298 13453 sys_catalog.cc:455] T 00000000000000000000000000000000 P 794f593770bb429487ad0ec805dae6bf [sys.catalog]: SysCatalogTable state changed. Reason: New leader 794f593770bb429487ad0ec805dae6bf. Latest consensus state: current_term: 1 leader_uuid: "794f593770bb429487ad0ec805dae6bf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "794f593770bb429487ad0ec805dae6bf" member_type: VOTER last_known_addr { host: "127.11.102.62" port: 39839 } } }
I20260620 06:06:51.625427 13453 sys_catalog.cc:458] T 00000000000000000000000000000000 P 794f593770bb429487ad0ec805dae6bf [sys.catalog]: This master's current role is: LEADER
I20260620 06:06:51.625334 13452 sys_catalog.cc:455] T 00000000000000000000000000000000 P 794f593770bb429487ad0ec805dae6bf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "794f593770bb429487ad0ec805dae6bf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "794f593770bb429487ad0ec805dae6bf" member_type: VOTER last_known_addr { host: "127.11.102.62" port: 39839 } } }
I20260620 06:06:51.625494 13452 sys_catalog.cc:458] T 00000000000000000000000000000000 P 794f593770bb429487ad0ec805dae6bf [sys.catalog]: This master's current role is: LEADER
I20260620 06:06:51.626433 13460 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 06:06:51.628961 13460 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 06:06:51.634044 13460 catalog_manager.cc:1347] Generated new cluster ID: 53631733faba4b85862a60eeae77a688
I20260620 06:06:51.634109 13460 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 06:06:51.651691 13460 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 06:06:51.652953 13460 catalog_manager.cc:1504] Loading token signing keys...
I20260620 06:06:51.668229 13460 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 794f593770bb429487ad0ec805dae6bf: Generated new TSK 0
I20260620 06:06:51.669032 13460 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.11.102.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.102.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.102.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781935603120653-11672-0/krb5kdc/kudu_127.11.102.1.keytab.
Entry for principal kudu/127.11.102.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781935603120653-11672-0/krb5kdc/kudu_127.11.102.1.keytab.
I20260620 06:06:51.698134 11672 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9HMp43/build/debug/bin/kudu
/tmp/dist-test-task9HMp43/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781935603120653-11672-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781935603120653-11672-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781935603120653-11672-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781935603120653-11672-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.102.1:0
--local_ip_for_outbound_sockets=127.11.102.1
--webserver_interface=127.11.102.1
--webserver_port=0
--tserver_master_addrs=127.11.102.62:39839
--keytab_file=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781935603120653-11672-0/krb5kdc/kudu_127.11.102.1.keytab
--principal=kudu/127.11.102.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.102.20:33411
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781935603120653-11672-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781935603120653-11672-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781935603120653-11672-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 06:06:51.806972 13472 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:06:51.807224 13472 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:06:51.807268 13472 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:06:51.811169 13472 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:06:51.811219 13472 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:06:51.811259 13472 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260620 06:06:51.811316 13472 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 06:06:51.811373 13472 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.102.1
I20260620 06:06:51.816350 13472 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.102.20:33411
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781935603120653-11672-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781935603120653-11672-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781935603120653-11672-0/krb5kdc/kudu_127.11.102.1.keytab
--principal=kudu/127.11.102.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.102.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781935603120653-11672-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.11.102.1
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.11.102.62:39839
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13472
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.102.1
--log_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781935603120653-11672-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 05:28:48 UTC on ccycloud.quasar-fixhms.root.comops.site
build id 4457020
I20260620 06:06:51.817283 13472 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:06:51.818254 13472 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 06:06:51.827495 13478 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:06:51.827497 13480 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:06:51.827896 13477 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:06:51.827948 13472 server_base.cc:1034] running on GCE node
I20260620 06:06:51.828419 13472 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:06:51.829178 13472 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:06:51.830363 13472 hybrid_clock.cc:648] HybridClock initialized: now 1781935611830326 us; error 42 us; skew 500 ppm
Jun 20 06:06:51 dist-test-slave-9hm5 krb5kdc[13355](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935611, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.102.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 06:06:51.833119 13472 init.cc:377] Logged in from keytab as kudu/127.11.102.1@KRBTEST.COM (short username kudu)
I20260620 06:06:51.833971 13472 webserver.cc:460] Webserver started at http://127.11.102.1:41933/ using document root <none> and password file <none>
I20260620 06:06:51.834501 13472 fs_manager.cc:359] Metadata directory not provided
I20260620 06:06:51.834559 13472 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:06:51.834779 13472 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 06:06:51.836760 13472 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781935603120653-11672-0/minicluster-data/ts-0/data/instance:
uuid: "5124d225ee724a998f64980e62a4dd51"
format_stamp: "Formatted at 2026-06-20 06:06:51 on dist-test-slave-9hm5"
server_key: "00004f0ae1a49ab33f5bb2c4f9d952a0"
server_key_iv: "f88d8b14ec157ef7c852c9159cb8e96b"
server_key_version: "encryptionkey@0"
I20260620 06:06:51.837299 13472 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781935603120653-11672-0/minicluster-data/ts-0/wal/instance:
uuid: "5124d225ee724a998f64980e62a4dd51"
format_stamp: "Formatted at 2026-06-20 06:06:51 on dist-test-slave-9hm5"
server_key: "00004f0ae1a49ab33f5bb2c4f9d952a0"
server_key_iv: "f88d8b14ec157ef7c852c9159cb8e96b"
server_key_version: "encryptionkey@0"
I20260620 06:06:51.840794 13472 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 06:06:51.842978 13487 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:06:51.844173 13472 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 06:06:51.844286 13472 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781935603120653-11672-0/minicluster-data/ts-0/data,/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781935603120653-11672-0/minicluster-data/ts-0/wal
uuid: "5124d225ee724a998f64980e62a4dd51"
format_stamp: "Formatted at 2026-06-20 06:06:51 on dist-test-slave-9hm5"
server_key: "00004f0ae1a49ab33f5bb2c4f9d952a0"
server_key_iv: "f88d8b14ec157ef7c852c9159cb8e96b"
server_key_version: "encryptionkey@0"
I20260620 06:06:51.844391 13472 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781935603120653-11672-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781935603120653-11672-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781935603120653-11672-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:06:51.861320 13472 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:06:51.864568 13472 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:06:51.864805 13472 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:06:51.866228 13472 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 06:06:51.866276 13472 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 06:06:51.866313 13472 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 06:06:51.866372 13472 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 06:06:51 dist-test-slave-9hm5 krb5kdc[13355](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935611, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.102.1@KRBTEST.COM for kudu/127.11.102.62@KRBTEST.COM
I20260620 06:06:51.877279 13472 rpc_server.cc:307] RPC server started. Bound to: 127.11.102.1:33471
I20260620 06:06:51.878154 13472 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781935603120653-11672-0/minicluster-data/ts-0/data/info.pb
I20260620 06:06:51.878813 13602 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.102.1:33471 every 8 connection(s)
I20260620 06:06:51.880486 13449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:06:51.867152 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.102.1:43259 (local address 127.11.102.62:39839)
0620 06:06:51.867281 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0620 06:06:51.867285 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 06:06:51.868378 (+ 1093us) server_negotiation.cc:407] Connection header received
0620 06:06:51.869510 (+ 1132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:06:51.869514 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:06:51.869584 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:06:51.869689 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:06:51.871336 (+ 1647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:06:51.872131 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:06:51.873485 (+ 1354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:06:51.873743 (+ 258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:06:51.873878 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:06:51.876641 (+ 2763us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:06:51.876644 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:06:51.876647 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:06:51.876674 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:06:51.878323 (+ 1649us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:06:51.878936 (+ 613us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:06:51.878942 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:06:51.878944 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:06:51.879056 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:06:51.879402 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:06:51.879405 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:06:51.879407 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:06:51.879587 (+ 180us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:06:51.879710 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0620 06:06:51.880191 (+ 481us) server_negotiation.cc:299] Negotiation successful
0620 06:06:51.880313 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260620 06:06:51.881217 13496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:06:51.867755 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.102.62:39839 (local address 127.11.102.1:43259)
0620 06:06:51.868230 (+ 475us) negotiation.cc:107] Waiting for socket to connect
0620 06:06:51.868272 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0620 06:06:51.869254 (+ 982us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:06:51.869904 (+ 650us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:06:51.869914 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:06:51.870354 (+ 440us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:06:51.871128 (+ 774us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:06:51.871155 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:06:51.872261 (+ 1106us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:06:51.872265 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:06:51.873365 (+ 1100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:06:51.873375 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:06:51.873917 (+ 542us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:06:51.873922 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:06:51.874133 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:06:51.874747 (+ 614us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:06:51.874762 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:06:51.876474 (+ 1712us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:06:51.878440 (+ 1966us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:06:51.878446 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:06:51.878460 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:06:51.878773 (+ 313us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:06:51.879148 (+ 375us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:06:51.879151 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:06:51.879153 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:06:51.879286 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:06:51.879718 (+ 432us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:06:51.879726 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:06:51.879957 (+ 231us) client_negotiation.cc:769] Sending connection context
0620 06:06:51.880136 (+ 179us) client_negotiation.cc:240] Negotiation successful
0620 06:06:51.880447 (+ 311us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":291,"thread_start_us":139,"threads_started":1}
I20260620 06:06:51.882563 13603 heartbeater.cc:347] Connected to a master server at 127.11.102.62:39839
I20260620 06:06:51.882716 13603 heartbeater.cc:464] Registering TS with master...
I20260620 06:06:51.883206 13603 heartbeater.cc:511] Master 127.11.102.62:39839 requested a full tablet report, sending...
I20260620 06:06:51.884776 13397 ts_manager.cc:194] Registered new tserver with Master: 5124d225ee724a998f64980e62a4dd51 (127.11.102.1:33471)
I20260620 06:06:51.885843 13397 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.102.1@KRBTEST.COM'} at 127.11.102.1:43259
I20260620 06:06:51.886504 11672 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9HMp43/build/debug/bin/kudu as pid 13472
I20260620 06:06:51.886646 11672 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781935603120653-11672-0/minicluster-data/ts-0/wal/instance
I20260620 06:06:51.886929 11672 external_mini_cluster.cc:1428] Setting key 2a2a6520cb8eb099157198eed3f3788a
I20260620 06:06:51.894161 13449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:06:51.884626 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.102.1:56383 (local address 127.11.102.62:39839)
0620 06:06:51.884744 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0620 06:06:51.884747 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 06:06:51.887491 (+ 2744us) server_negotiation.cc:407] Connection header received
0620 06:06:51.887669 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:06:51.887672 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:06:51.887714 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:06:51.887795 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:06:51.888261 (+ 466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:06:51.888792 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:06:51.889590 (+ 798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:06:51.889815 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:06:51.889921 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:06:51.891005 (+ 1084us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:06:51.891008 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:06:51.891010 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:06:51.891030 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:06:51.892500 (+ 1470us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:06:51.893000 (+ 500us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:06:51.893003 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:06:51.893004 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:06:51.893077 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:06:51.893382 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:06:51.893385 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:06:51.893386 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:06:51.893553 (+ 167us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:06:51.893626 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0620 06:06:51.893969 (+ 343us) server_negotiation.cc:299] Negotiation successful
0620 06:06:51.894057 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260620 06:06:51.894306 13609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:06:51.884558 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.102.62:39839 (local address 127.11.102.1:56383)
0620 06:06:51.887418 (+ 2860us) negotiation.cc:107] Waiting for socket to connect
0620 06:06:51.887433 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 06:06:51.887527 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:06:51.887830 (+ 303us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:06:51.887833 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:06:51.888039 (+ 206us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:06:51.888147 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:06:51.888154 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:06:51.888920 (+ 766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:06:51.888923 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:06:51.889475 (+ 552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:06:51.889482 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:06:51.889923 (+ 441us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:06:51.889926 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:06:51.890021 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:06:51.890534 (+ 513us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:06:51.890549 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:06:51.890883 (+ 334us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:06:51.892628 (+ 1745us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:06:51.892634 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:06:51.892639 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:06:51.892899 (+ 260us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:06:51.893175 (+ 276us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:06:51.893180 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:06:51.893184 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:06:51.893286 (+ 102us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:06:51.893649 (+ 363us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:06:51.893654 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:06:51.893765 (+ 111us) client_negotiation.cc:769] Sending connection context
0620 06:06:51.893910 (+ 145us) client_negotiation.cc:240] Negotiation successful
0620 06:06:51.894079 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2773,"thread_start_us":85,"threads_started":1}
WARNING: no policy specified for kudu/127.11.102.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.102.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.102.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781935603120653-11672-0/krb5kdc/kudu_127.11.102.2.keytab.
Entry for principal kudu/127.11.102.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781935603120653-11672-0/krb5kdc/kudu_127.11.102.2.keytab.
I20260620 06:06:51.914354 11672 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9HMp43/build/debug/bin/kudu
/tmp/dist-test-task9HMp43/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781935603120653-11672-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781935603120653-11672-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781935603120653-11672-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781935603120653-11672-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.102.2:0
--local_ip_for_outbound_sockets=127.11.102.2
--webserver_interface=127.11.102.2
--webserver_port=0
--tserver_master_addrs=127.11.102.62:39839
--keytab_file=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781935603120653-11672-0/krb5kdc/kudu_127.11.102.2.keytab
--principal=kudu/127.11.102.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.102.20:33411
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781935603120653-11672-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781935603120653-11672-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781935603120653-11672-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 06:06:52.021509 13612 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:06:52.021778 13612 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:06:52.021862 13612 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:06:52.026269 13612 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:06:52.026346 13612 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:06:52.026441 13612 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260620 06:06:52.026477 13612 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 06:06:52.026538 13612 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.102.2
I20260620 06:06:52.031764 13612 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.102.20:33411
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781935603120653-11672-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781935603120653-11672-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781935603120653-11672-0/krb5kdc/kudu_127.11.102.2.keytab
--principal=kudu/127.11.102.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.102.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781935603120653-11672-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.11.102.2
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.11.102.62:39839
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13612
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.102.2
--log_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781935603120653-11672-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 05:28:48 UTC on ccycloud.quasar-fixhms.root.comops.site
build id 4457020
I20260620 06:06:52.032694 13612 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:06:52.033634 13612 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 06:06:52.042959 13620 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:06:52.042955 13617 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:06:52.042996 13618 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:06:52.043278 13612 server_base.cc:1034] running on GCE node
I20260620 06:06:52.043751 13612 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:06:52.044302 13612 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:06:52.045459 13612 hybrid_clock.cc:648] HybridClock initialized: now 1781935612045439 us; error 31 us; skew 500 ppm
Jun 20 06:06:52 dist-test-slave-9hm5 krb5kdc[13355](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935612, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.102.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 06:06:52.048190 13612 init.cc:377] Logged in from keytab as kudu/127.11.102.2@KRBTEST.COM (short username kudu)
I20260620 06:06:52.048985 13612 webserver.cc:460] Webserver started at http://127.11.102.2:45913/ using document root <none> and password file <none>
I20260620 06:06:52.049537 13612 fs_manager.cc:359] Metadata directory not provided
I20260620 06:06:52.049626 13612 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:06:52.049849 13612 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 06:06:52.051919 13612 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781935603120653-11672-0/minicluster-data/ts-1/data/instance:
uuid: "73042c6c0e8a4affa6e460bb19d7c5c9"
format_stamp: "Formatted at 2026-06-20 06:06:52 on dist-test-slave-9hm5"
server_key: "4130baa2e924fd7dfe4a58e51ca02b1d"
server_key_iv: "5cb0d9edca9028f7e16b572d5c0df9da"
server_key_version: "encryptionkey@0"
I20260620 06:06:52.052477 13612 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781935603120653-11672-0/minicluster-data/ts-1/wal/instance:
uuid: "73042c6c0e8a4affa6e460bb19d7c5c9"
format_stamp: "Formatted at 2026-06-20 06:06:52 on dist-test-slave-9hm5"
server_key: "4130baa2e924fd7dfe4a58e51ca02b1d"
server_key_iv: "5cb0d9edca9028f7e16b572d5c0df9da"
server_key_version: "encryptionkey@0"
I20260620 06:06:52.055959 13612 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260620 06:06:52.058269 13627 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:06:52.059528 13612 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 06:06:52.059672 13612 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781935603120653-11672-0/minicluster-data/ts-1/data,/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781935603120653-11672-0/minicluster-data/ts-1/wal
uuid: "73042c6c0e8a4affa6e460bb19d7c5c9"
format_stamp: "Formatted at 2026-06-20 06:06:52 on dist-test-slave-9hm5"
server_key: "4130baa2e924fd7dfe4a58e51ca02b1d"
server_key_iv: "5cb0d9edca9028f7e16b572d5c0df9da"
server_key_version: "encryptionkey@0"
I20260620 06:06:52.059780 13612 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781935603120653-11672-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781935603120653-11672-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781935603120653-11672-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:06:52.070571 13612 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:06:52.073324 13612 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:06:52.073534 13612 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:06:52.074807 13612 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 06:06:52.074888 13612 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 06:06:52.074976 13612 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 06:06:52.075029 13612 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 06:06:52 dist-test-slave-9hm5 krb5kdc[13355](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935612, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.102.2@KRBTEST.COM for kudu/127.11.102.62@KRBTEST.COM
I20260620 06:06:52.085536 13612 rpc_server.cc:307] RPC server started. Bound to: 127.11.102.2:43331
I20260620 06:06:52.085605 13742 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.102.2:43331 every 8 connection(s)
I20260620 06:06:52.086571 13612 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781935603120653-11672-0/minicluster-data/ts-1/data/info.pb
I20260620 06:06:52.088698 13449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:06:52.075702 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.102.2:42355 (local address 127.11.102.62:39839)
0620 06:06:52.075852 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0620 06:06:52.075857 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 06:06:52.076666 (+ 809us) server_negotiation.cc:407] Connection header received
0620 06:06:52.077596 (+ 930us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:06:52.077599 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:06:52.077657 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:06:52.077719 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:06:52.079400 (+ 1681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:06:52.080332 (+ 932us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:06:52.082067 (+ 1735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:06:52.082477 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:06:52.082618 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:06:52.085219 (+ 2601us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:06:52.085225 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:06:52.085229 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:06:52.085267 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:06:52.086829 (+ 1562us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:06:52.087356 (+ 527us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:06:52.087360 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:06:52.087362 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:06:52.087447 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:06:52.087821 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:06:52.087824 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:06:52.087825 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:06:52.087994 (+ 169us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:06:52.088070 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0620 06:06:52.088490 (+ 420us) server_negotiation.cc:299] Negotiation successful
0620 06:06:52.088582 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260620 06:06:52.089460 13636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:06:52.076092 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.102.62:39839 (local address 127.11.102.2:42355)
0620 06:06:52.076508 (+ 416us) negotiation.cc:107] Waiting for socket to connect
0620 06:06:52.076553 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0620 06:06:52.077362 (+ 809us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:06:52.078153 (+ 791us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:06:52.078162 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:06:52.078651 (+ 489us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:06:52.079257 (+ 606us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:06:52.079279 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:06:52.080465 (+ 1186us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:06:52.080470 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:06:52.081921 (+ 1451us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:06:52.081932 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:06:52.082648 (+ 716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:06:52.082652 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:06:52.082856 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:06:52.083433 (+ 577us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:06:52.083453 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:06:52.085045 (+ 1592us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:06:52.086937 (+ 1892us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:06:52.086943 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:06:52.086956 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:06:52.087266 (+ 310us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:06:52.087542 (+ 276us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:06:52.087545 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:06:52.087546 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:06:52.087738 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:06:52.088079 (+ 341us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:06:52.088084 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:06:52.088290 (+ 206us) client_negotiation.cc:769] Sending connection context
0620 06:06:52.088432 (+ 142us) client_negotiation.cc:240] Negotiation successful
0620 06:06:52.088656 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":236,"thread_start_us":125,"threads_started":1}
I20260620 06:06:52.089902 11672 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9HMp43/build/debug/bin/kudu as pid 13612
I20260620 06:06:52.090020 11672 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781935603120653-11672-0/minicluster-data/ts-1/wal/instance
I20260620 06:06:52.090341 11672 external_mini_cluster.cc:1428] Setting key 6b1a9088c30ed757d46072cf368a0137
I20260620 06:06:52.090775 13743 heartbeater.cc:347] Connected to a master server at 127.11.102.62:39839
I20260620 06:06:52.090920 13743 heartbeater.cc:464] Registering TS with master...
I20260620 06:06:52.091439 13743 heartbeater.cc:511] Master 127.11.102.62:39839 requested a full tablet report, sending...
I20260620 06:06:52.094079 13400 ts_manager.cc:194] Registered new tserver with Master: 73042c6c0e8a4affa6e460bb19d7c5c9 (127.11.102.2:43331)
I20260620 06:06:52.094643 13400 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.102.2@KRBTEST.COM'} at 127.11.102.2:42355
WARNING: no policy specified for kudu/127.11.102.3@KRBTEST.COM; defaulting to no policy
I20260620 06:06:52.100112 13449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:06:52.092804 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.102.2:44985 (local address 127.11.102.62:39839)
0620 06:06:52.092932 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0620 06:06:52.092934 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 06:06:52.093248 (+ 314us) server_negotiation.cc:407] Connection header received
0620 06:06:52.093356 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:06:52.093361 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:06:52.093411 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:06:52.093500 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:06:52.094007 (+ 507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:06:52.094683 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:06:52.095653 (+ 970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:06:52.095868 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:06:52.095978 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:06:52.097027 (+ 1049us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:06:52.097031 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:06:52.097033 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:06:52.097062 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:06:52.098452 (+ 1390us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:06:52.098871 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:06:52.098876 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:06:52.098878 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:06:52.098951 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:06:52.099218 (+ 267us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:06:52.099223 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:06:52.099225 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:06:52.099428 (+ 203us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:06:52.099535 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0620 06:06:52.099832 (+ 297us) server_negotiation.cc:299] Negotiation successful
0620 06:06:52.099944 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260620 06:06:52.100128 13749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:06:52.092763 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.102.62:39839 (local address 127.11.102.2:44985)
0620 06:06:52.093153 (+ 390us) negotiation.cc:107] Waiting for socket to connect
0620 06:06:52.093171 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 06:06:52.093275 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:06:52.093527 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:06:52.093530 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:06:52.093765 (+ 235us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:06:52.093888 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:06:52.093896 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:06:52.094815 (+ 919us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:06:52.094818 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:06:52.095503 (+ 685us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:06:52.095511 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:06:52.095977 (+ 466us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:06:52.095981 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:06:52.096068 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:06:52.096567 (+ 499us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:06:52.096581 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:06:52.096899 (+ 318us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:06:52.098571 (+ 1672us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:06:52.098574 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:06:52.098576 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:06:52.098775 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:06:52.099040 (+ 265us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:06:52.099044 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:06:52.099046 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:06:52.099131 (+ 85us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:06:52.099524 (+ 393us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:06:52.099527 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:06:52.099641 (+ 114us) client_negotiation.cc:769] Sending connection context
0620 06:06:52.099777 (+ 136us) client_negotiation.cc:240] Negotiation successful
0620 06:06:52.099947 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":297,"mutex_wait_us":3,"thread_start_us":88,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.102.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.102.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781935603120653-11672-0/krb5kdc/kudu_127.11.102.3.keytab.
Entry for principal kudu/127.11.102.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781935603120653-11672-0/krb5kdc/kudu_127.11.102.3.keytab.
I20260620 06:06:52.118000 11672 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9HMp43/build/debug/bin/kudu
/tmp/dist-test-task9HMp43/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781935603120653-11672-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781935603120653-11672-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781935603120653-11672-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781935603120653-11672-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.102.3:0
--local_ip_for_outbound_sockets=127.11.102.3
--webserver_interface=127.11.102.3
--webserver_port=0
--tserver_master_addrs=127.11.102.62:39839
--keytab_file=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781935603120653-11672-0/krb5kdc/kudu_127.11.102.3.keytab
--principal=kudu/127.11.102.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.102.20:33411
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781935603120653-11672-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781935603120653-11672-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781935603120653-11672-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 06:06:52.227069 13752 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:06:52.227298 13752 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:06:52.227341 13752 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:06:52.231262 13752 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:06:52.231313 13752 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:06:52.231352 13752 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260620 06:06:52.231413 13752 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 06:06:52.231469 13752 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.102.3
I20260620 06:06:52.236521 13752 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.102.20:33411
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781935603120653-11672-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781935603120653-11672-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781935603120653-11672-0/krb5kdc/kudu_127.11.102.3.keytab
--principal=kudu/127.11.102.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.102.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781935603120653-11672-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.11.102.3
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.11.102.62:39839
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13752
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.102.3
--log_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781935603120653-11672-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 05:28:48 UTC on ccycloud.quasar-fixhms.root.comops.site
build id 4457020
I20260620 06:06:52.237443 13752 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:06:52.238405 13752 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 06:06:52.247493 13758 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:06:52.247538 13752 server_base.cc:1034] running on GCE node
W20260620 06:06:52.247562 13757 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:06:52.247733 13760 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:06:52.248102 13752 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:06:52.248761 13752 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:06:52.249940 13752 hybrid_clock.cc:648] HybridClock initialized: now 1781935612249911 us; error 43 us; skew 500 ppm
Jun 20 06:06:52 dist-test-slave-9hm5 krb5kdc[13355](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935612, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.102.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 06:06:52.252565 13752 init.cc:377] Logged in from keytab as kudu/127.11.102.3@KRBTEST.COM (short username kudu)
I20260620 06:06:52.253417 13752 webserver.cc:460] Webserver started at http://127.11.102.3:36141/ using document root <none> and password file <none>
I20260620 06:06:52.253961 13752 fs_manager.cc:359] Metadata directory not provided
I20260620 06:06:52.254051 13752 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:06:52.254276 13752 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 06:06:52.256263 13752 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781935603120653-11672-0/minicluster-data/ts-2/data/instance:
uuid: "e61acac2279e47ffbb9af68a1df4d31d"
format_stamp: "Formatted at 2026-06-20 06:06:52 on dist-test-slave-9hm5"
server_key: "df8d884761bc0a482dba20d23b84ca50"
server_key_iv: "fc8d9da8357381185696ba930f057ad7"
server_key_version: "encryptionkey@0"
I20260620 06:06:52.256805 13752 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781935603120653-11672-0/minicluster-data/ts-2/wal/instance:
uuid: "e61acac2279e47ffbb9af68a1df4d31d"
format_stamp: "Formatted at 2026-06-20 06:06:52 on dist-test-slave-9hm5"
server_key: "df8d884761bc0a482dba20d23b84ca50"
server_key_iv: "fc8d9da8357381185696ba930f057ad7"
server_key_version: "encryptionkey@0"
I20260620 06:06:52.260329 13752 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260620 06:06:52.262559 13767 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:06:52.263882 13752 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 06:06:52.263969 13752 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781935603120653-11672-0/minicluster-data/ts-2/data,/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781935603120653-11672-0/minicluster-data/ts-2/wal
uuid: "e61acac2279e47ffbb9af68a1df4d31d"
format_stamp: "Formatted at 2026-06-20 06:06:52 on dist-test-slave-9hm5"
server_key: "df8d884761bc0a482dba20d23b84ca50"
server_key_iv: "fc8d9da8357381185696ba930f057ad7"
server_key_version: "encryptionkey@0"
I20260620 06:06:52.264047 13752 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781935603120653-11672-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781935603120653-11672-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781935603120653-11672-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:06:52.276506 13752 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:06:52.279135 13752 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:06:52.279278 13752 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:06:52.280675 13752 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 06:06:52.280732 13752 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 06:06:52.280772 13752 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 06:06:52.280787 13752 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 06:06:52 dist-test-slave-9hm5 krb5kdc[13355](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935612, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.102.3@KRBTEST.COM for kudu/127.11.102.62@KRBTEST.COM
I20260620 06:06:52.291847 13752 rpc_server.cc:307] RPC server started. Bound to: 127.11.102.3:38435
I20260620 06:06:52.291890 13882 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.102.3:38435 every 8 connection(s)
I20260620 06:06:52.292708 13752 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781935603120653-11672-0/minicluster-data/ts-2/data/info.pb
I20260620 06:06:52.293840 11672 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9HMp43/build/debug/bin/kudu as pid 13752
I20260620 06:06:52.293965 11672 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781935603120653-11672-0/minicluster-data/ts-2/wal/instance
I20260620 06:06:52.294222 11672 external_mini_cluster.cc:1428] Setting key f5a7a26d4b96206207900af811aee07a
I20260620 06:06:52.294569 13449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:06:52.281462 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.102.3:35339 (local address 127.11.102.62:39839)
0620 06:06:52.281596 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0620 06:06:52.281600 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 06:06:52.282647 (+ 1047us) server_negotiation.cc:407] Connection header received
0620 06:06:52.283806 (+ 1159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:06:52.283810 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:06:52.283872 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:06:52.283958 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:06:52.285499 (+ 1541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:06:52.286198 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:06:52.287342 (+ 1144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:06:52.287653 (+ 311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:06:52.287759 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:06:52.290648 (+ 2889us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:06:52.290652 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:06:52.290654 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:06:52.290681 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:06:52.292295 (+ 1614us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:06:52.292907 (+ 612us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:06:52.292916 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:06:52.292918 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:06:52.293015 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:06:52.293405 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:06:52.293411 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:06:52.293413 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:06:52.293660 (+ 247us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:06:52.293768 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0620 06:06:52.294284 (+ 516us) server_negotiation.cc:299] Negotiation successful
0620 06:06:52.294402 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260620 06:06:52.295363 13776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:06:52.281999 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.102.62:39839 (local address 127.11.102.3:35339)
0620 06:06:52.282486 (+ 487us) negotiation.cc:107] Waiting for socket to connect
0620 06:06:52.282526 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0620 06:06:52.283510 (+ 984us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:06:52.284171 (+ 661us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:06:52.284180 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:06:52.284670 (+ 490us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:06:52.285314 (+ 644us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:06:52.285332 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:06:52.286322 (+ 990us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:06:52.286327 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:06:52.287210 (+ 883us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:06:52.287219 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:06:52.287800 (+ 581us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:06:52.287804 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:06:52.288028 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:06:52.288616 (+ 588us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:06:52.288639 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:06:52.290476 (+ 1837us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:06:52.292405 (+ 1929us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:06:52.292412 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:06:52.292429 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:06:52.292723 (+ 294us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:06:52.293126 (+ 403us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:06:52.293129 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:06:52.293132 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:06:52.293287 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:06:52.293774 (+ 487us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:06:52.293783 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:06:52.294036 (+ 253us) client_negotiation.cc:769] Sending connection context
0620 06:06:52.294238 (+ 202us) client_negotiation.cc:240] Negotiation successful
0620 06:06:52.294535 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":310,"thread_start_us":142,"threads_started":1}
I20260620 06:06:52.296568 13883 heartbeater.cc:347] Connected to a master server at 127.11.102.62:39839
I20260620 06:06:52.296706 13883 heartbeater.cc:464] Registering TS with master...
I20260620 06:06:52.297241 13883 heartbeater.cc:511] Master 127.11.102.62:39839 requested a full tablet report, sending...
I20260620 06:06:52.298218 13400 ts_manager.cc:194] Registered new tserver with Master: e61acac2279e47ffbb9af68a1df4d31d (127.11.102.3:38435)
I20260620 06:06:52.298928 13400 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.102.3@KRBTEST.COM'} at 127.11.102.3:35339
I20260620 06:06:52.304987 13889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:06:52.298553 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.102.62:39839 (local address 127.11.102.3:50409)
0620 06:06:52.298856 (+ 303us) negotiation.cc:107] Waiting for socket to connect
0620 06:06:52.298868 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 06:06:52.298973 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:06:52.299196 (+ 223us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:06:52.299199 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:06:52.299392 (+ 193us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:06:52.299515 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:06:52.299523 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:06:52.300278 (+ 755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:06:52.300281 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:06:52.301042 (+ 761us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:06:52.301050 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:06:52.301448 (+ 398us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:06:52.301450 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:06:52.301514 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:06:52.301996 (+ 482us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:06:52.302015 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:06:52.302297 (+ 282us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:06:52.303826 (+ 1529us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:06:52.303829 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:06:52.303830 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:06:52.304015 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:06:52.304244 (+ 229us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:06:52.304246 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:06:52.304247 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:06:52.304320 (+ 73us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:06:52.304622 (+ 302us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:06:52.304624 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:06:52.304692 (+ 68us) client_negotiation.cc:769] Sending connection context
0620 06:06:52.304769 (+ 77us) client_negotiation.cc:240] Negotiation successful
0620 06:06:52.304865 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":240,"thread_start_us":80,"threads_started":1}
I20260620 06:06:52.305007 13449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:06:52.298633 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.102.3:50409 (local address 127.11.102.62:39839)
0620 06:06:52.298726 (+ 93us) server_negotiation.cc:206] Beginning negotiation
0620 06:06:52.298729 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 06:06:52.298937 (+ 208us) server_negotiation.cc:407] Connection header received
0620 06:06:52.299048 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:06:52.299051 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:06:52.299098 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:06:52.299190 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:06:52.299677 (+ 487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:06:52.300154 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:06:52.301146 (+ 992us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:06:52.301349 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:06:52.301430 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:06:52.302414 (+ 984us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:06:52.302417 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:06:52.302419 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:06:52.302438 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:06:52.303710 (+ 1272us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:06:52.304098 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:06:52.304101 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:06:52.304102 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:06:52.304168 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:06:52.304393 (+ 225us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:06:52.304395 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:06:52.304396 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:06:52.304540 (+ 144us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:06:52.304612 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0620 06:06:52.304822 (+ 210us) server_negotiation.cc:299] Negotiation successful
0620 06:06:52.304904 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260620 06:06:52.307997 11672 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 20 06:06:52 dist-test-slave-9hm5 krb5kdc[13355](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935612, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 20 06:06:52 dist-test-slave-9hm5 krb5kdc[13355](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935612, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.102.62@KRBTEST.COM
I20260620 06:06:52.332791 13449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:06:52.324791 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57814 (local address 127.11.102.62:39839)
0620 06:06:52.324911 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0620 06:06:52.324915 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 06:06:52.324926 (+ 11us) server_negotiation.cc:407] Connection header received
0620 06:06:52.324961 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:06:52.324963 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:06:52.324999 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:06:52.325097 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:06:52.325827 (+ 730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:06:52.326423 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:06:52.327345 (+ 922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:06:52.327550 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:06:52.327777 (+ 227us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:06:52.330075 (+ 2298us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:06:52.330078 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:06:52.330080 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:06:52.330103 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:06:52.331322 (+ 1219us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:06:52.331757 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:06:52.331759 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:06:52.331761 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:06:52.331836 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:06:52.332064 (+ 228us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:06:52.332066 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:06:52.332067 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:06:52.332222 (+ 155us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:06:52.332321 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0620 06:06:52.332487 (+ 166us) server_negotiation.cc:299] Negotiation successful
0620 06:06:52.332579 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260620 06:06:52.335438 13400 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57814:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 06:06:52.338536 13400 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 06:06:52.352952 13905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:06:52.347842 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39920 (local address 127.11.102.3:38435)
0620 06:06:52.348106 (+ 264us) server_negotiation.cc:206] Beginning negotiation
0620 06:06:52.348121 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 06:06:52.348141 (+ 20us) server_negotiation.cc:407] Connection header received
0620 06:06:52.348209 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:06:52.348214 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:06:52.348378 (+ 164us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:06:52.348511 (+ 133us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 06:06:52.348686 (+ 175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:06:52.350335 (+ 1649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:06:52.351391 (+ 1056us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:06:52.352229 (+ 838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:06:52.352342 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:06:52.352510 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0620 06:06:52.352701 (+ 191us) server_negotiation.cc:299] Negotiation successful
0620 06:06:52.352793 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":179,"thread_start_us":99,"threads_started":1}
I20260620 06:06:52.352946 13904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:06:52.347481 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.102.3:38435 (local address 127.0.0.1:39920)
0620 06:06:52.347774 (+ 293us) negotiation.cc:107] Waiting for socket to connect
0620 06:06:52.347788 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 06:06:52.347873 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:06:52.348491 (+ 618us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:06:52.348493 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:06:52.348504 (+ 11us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 06:06:52.348576 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:06:52.348581 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:06:52.350422 (+ 1841us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:06:52.350427 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:06:52.351272 (+ 845us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:06:52.351280 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:06:52.352354 (+ 1074us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:06:52.352361 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:06:52.352506 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:06:52.352546 (+ 40us) client_negotiation.cc:769] Sending connection context
0620 06:06:52.352662 (+ 116us) client_negotiation.cc:240] Negotiation successful
0620 06:06:52.352779 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":235,"mutex_wait_us":71,"thread_start_us":274,"threads_started":1}
I20260620 06:06:52.353264 13901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:06:52.346443 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.102.2:43331 (local address 127.0.0.1:50158)
0620 06:06:52.347502 (+ 1059us) negotiation.cc:107] Waiting for socket to connect
0620 06:06:52.347514 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 06:06:52.347724 (+ 210us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:06:52.348375 (+ 651us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:06:52.348381 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:06:52.348417 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 06:06:52.348827 (+ 410us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:06:52.348833 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:06:52.349837 (+ 1004us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:06:52.349843 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:06:52.351459 (+ 1616us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:06:52.351469 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:06:52.352668 (+ 1199us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:06:52.352671 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:06:52.352759 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:06:52.352781 (+ 22us) client_negotiation.cc:769] Sending connection context
0620 06:06:52.353010 (+ 229us) client_negotiation.cc:240] Negotiation successful
0620 06:06:52.353094 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1000,"mutex_wait_us":388,"spinlock_wait_cycles":6784,"thread_start_us":100,"threads_started":1}
I20260620 06:06:52.353479 13900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:06:52.346433 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.102.1:33471 (local address 127.0.0.1:56016)
0620 06:06:52.347099 (+ 666us) negotiation.cc:107] Waiting for socket to connect
0620 06:06:52.347150 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0620 06:06:52.347270 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:06:52.348375 (+ 1105us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:06:52.348381 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:06:52.348415 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 06:06:52.348566 (+ 151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:06:52.348580 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:06:52.349969 (+ 1389us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:06:52.349973 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:06:52.351459 (+ 1486us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:06:52.351470 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:06:52.352392 (+ 922us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:06:52.352395 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:06:52.353151 (+ 756us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:06:52.353187 (+ 36us) client_negotiation.cc:769] Sending connection context
0620 06:06:52.353271 (+ 84us) client_negotiation.cc:240] Negotiation successful
0620 06:06:52.353346 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":250,"mutex_wait_us":741,"spinlock_wait_cycles":5248,"thread_start_us":57,"threads_started":1}
I20260620 06:06:52.353695 13902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:06:52.346572 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56016 (local address 127.11.102.1:33471)
0620 06:06:52.347462 (+ 890us) server_negotiation.cc:206] Beginning negotiation
0620 06:06:52.347477 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 06:06:52.347500 (+ 23us) server_negotiation.cc:407] Connection header received
0620 06:06:52.347576 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:06:52.347584 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:06:52.348213 (+ 629us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:06:52.348361 (+ 148us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 06:06:52.348691 (+ 330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:06:52.349869 (+ 1178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:06:52.351698 (+ 1829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:06:52.352252 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:06:52.352501 (+ 249us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:06:52.352671 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0620 06:06:52.353320 (+ 649us) server_negotiation.cc:299] Negotiation successful
0620 06:06:52.353527 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":778,"thread_start_us":73,"threads_started":1}
I20260620 06:06:52.353758 13903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:06:52.346916 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50158 (local address 127.11.102.2:43331)
0620 06:06:52.347257 (+ 341us) server_negotiation.cc:206] Beginning negotiation
0620 06:06:52.347270 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 06:06:52.347593 (+ 323us) server_negotiation.cc:407] Connection header received
0620 06:06:52.347877 (+ 284us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:06:52.347883 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:06:52.348088 (+ 205us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:06:52.348191 (+ 103us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 06:06:52.348931 (+ 740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:06:52.349725 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:06:52.351577 (+ 1852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:06:52.352359 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:06:52.352450 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:06:52.352579 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0620 06:06:52.353416 (+ 837us) server_negotiation.cc:299] Negotiation successful
0620 06:06:52.353509 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":235,"thread_start_us":115,"threads_started":1}
I20260620 06:06:52.356233 13817 tablet_service.cc:1467] Processing CreateTablet for tablet ab6fcfff344d4968917c9d5991f20398 (DEFAULT_TABLE table=test-table [id=25eb0e3072b9489bb49f008916c8bc70]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 06:06:52.356235 13677 tablet_service.cc:1467] Processing CreateTablet for tablet ab6fcfff344d4968917c9d5991f20398 (DEFAULT_TABLE table=test-table [id=25eb0e3072b9489bb49f008916c8bc70]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 06:06:52.356235 13537 tablet_service.cc:1467] Processing CreateTablet for tablet ab6fcfff344d4968917c9d5991f20398 (DEFAULT_TABLE table=test-table [id=25eb0e3072b9489bb49f008916c8bc70]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 06:06:52.357146 13817 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ab6fcfff344d4968917c9d5991f20398. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 06:06:52.357172 13537 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ab6fcfff344d4968917c9d5991f20398. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 06:06:52.357172 13677 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ab6fcfff344d4968917c9d5991f20398. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 06:06:52.363365 13906 tablet_bootstrap.cc:492] T ab6fcfff344d4968917c9d5991f20398 P e61acac2279e47ffbb9af68a1df4d31d: Bootstrap starting.
I20260620 06:06:52.365280 13908 tablet_bootstrap.cc:492] T ab6fcfff344d4968917c9d5991f20398 P 5124d225ee724a998f64980e62a4dd51: Bootstrap starting.
I20260620 06:06:52.365677 13906 tablet_bootstrap.cc:654] T ab6fcfff344d4968917c9d5991f20398 P e61acac2279e47ffbb9af68a1df4d31d: Neither blocks nor log segments found. Creating new log.
I20260620 06:06:52.366407 13906 log.cc:824] T ab6fcfff344d4968917c9d5991f20398 P e61acac2279e47ffbb9af68a1df4d31d: Log is configured to *not* fsync() on all Append() calls
I20260620 06:06:52.366957 13908 tablet_bootstrap.cc:654] T ab6fcfff344d4968917c9d5991f20398 P 5124d225ee724a998f64980e62a4dd51: Neither blocks nor log segments found. Creating new log.
I20260620 06:06:52.367787 13908 log.cc:824] T ab6fcfff344d4968917c9d5991f20398 P 5124d225ee724a998f64980e62a4dd51: Log is configured to *not* fsync() on all Append() calls
I20260620 06:06:52.368000 13907 tablet_bootstrap.cc:492] T ab6fcfff344d4968917c9d5991f20398 P 73042c6c0e8a4affa6e460bb19d7c5c9: Bootstrap starting.
I20260620 06:06:52.368256 13906 tablet_bootstrap.cc:492] T ab6fcfff344d4968917c9d5991f20398 P e61acac2279e47ffbb9af68a1df4d31d: No bootstrap required, opened a new log
I20260620 06:06:52.368443 13906 ts_tablet_manager.cc:1397] T ab6fcfff344d4968917c9d5991f20398 P e61acac2279e47ffbb9af68a1df4d31d: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260620 06:06:52.369614 13908 tablet_bootstrap.cc:492] T ab6fcfff344d4968917c9d5991f20398 P 5124d225ee724a998f64980e62a4dd51: No bootstrap required, opened a new log
I20260620 06:06:52.369786 13908 ts_tablet_manager.cc:1397] T ab6fcfff344d4968917c9d5991f20398 P 5124d225ee724a998f64980e62a4dd51: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.003s
I20260620 06:06:52.370182 13907 tablet_bootstrap.cc:654] T ab6fcfff344d4968917c9d5991f20398 P 73042c6c0e8a4affa6e460bb19d7c5c9: Neither blocks nor log segments found. Creating new log.
I20260620 06:06:52.370914 13907 log.cc:824] T ab6fcfff344d4968917c9d5991f20398 P 73042c6c0e8a4affa6e460bb19d7c5c9: Log is configured to *not* fsync() on all Append() calls
I20260620 06:06:52.371439 13906 raft_consensus.cc:348] T ab6fcfff344d4968917c9d5991f20398 P e61acac2279e47ffbb9af68a1df4d31d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5124d225ee724a998f64980e62a4dd51" member_type: VOTER last_known_addr { host: "127.11.102.1" port: 33471 } } peers { permanent_uuid: "73042c6c0e8a4affa6e460bb19d7c5c9" member_type: VOTER last_known_addr { host: "127.11.102.2" port: 43331 } } peers { permanent_uuid: "e61acac2279e47ffbb9af68a1df4d31d" member_type: VOTER last_known_addr { host: "127.11.102.3" port: 38435 } }
I20260620 06:06:52.371744 13906 raft_consensus.cc:374] T ab6fcfff344d4968917c9d5991f20398 P e61acac2279e47ffbb9af68a1df4d31d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 06:06:52.371799 13906 raft_consensus.cc:729] T ab6fcfff344d4968917c9d5991f20398 P e61acac2279e47ffbb9af68a1df4d31d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e61acac2279e47ffbb9af68a1df4d31d, State: Initialized, Role: FOLLOWER
I20260620 06:06:52.372216 13906 consensus_queue.cc:260] T ab6fcfff344d4968917c9d5991f20398 P e61acac2279e47ffbb9af68a1df4d31d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5124d225ee724a998f64980e62a4dd51" member_type: VOTER last_known_addr { host: "127.11.102.1" port: 33471 } } peers { permanent_uuid: "73042c6c0e8a4affa6e460bb19d7c5c9" member_type: VOTER last_known_addr { host: "127.11.102.2" port: 43331 } } peers { permanent_uuid: "e61acac2279e47ffbb9af68a1df4d31d" member_type: VOTER last_known_addr { host: "127.11.102.3" port: 38435 } }
I20260620 06:06:52.372794 13907 tablet_bootstrap.cc:492] T ab6fcfff344d4968917c9d5991f20398 P 73042c6c0e8a4affa6e460bb19d7c5c9: No bootstrap required, opened a new log
I20260620 06:06:52.372808 13883 heartbeater.cc:503] Master 127.11.102.62:39839 was elected leader, sending a full tablet report...
I20260620 06:06:52.372960 13907 ts_tablet_manager.cc:1397] T ab6fcfff344d4968917c9d5991f20398 P 73042c6c0e8a4affa6e460bb19d7c5c9: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260620 06:06:52.373157 13906 ts_tablet_manager.cc:1428] T ab6fcfff344d4968917c9d5991f20398 P e61acac2279e47ffbb9af68a1df4d31d: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260620 06:06:52.373490 13908 raft_consensus.cc:348] T ab6fcfff344d4968917c9d5991f20398 P 5124d225ee724a998f64980e62a4dd51 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5124d225ee724a998f64980e62a4dd51" member_type: VOTER last_known_addr { host: "127.11.102.1" port: 33471 } } peers { permanent_uuid: "73042c6c0e8a4affa6e460bb19d7c5c9" member_type: VOTER last_known_addr { host: "127.11.102.2" port: 43331 } } peers { permanent_uuid: "e61acac2279e47ffbb9af68a1df4d31d" member_type: VOTER last_known_addr { host: "127.11.102.3" port: 38435 } }
I20260620 06:06:52.373741 13908 raft_consensus.cc:374] T ab6fcfff344d4968917c9d5991f20398 P 5124d225ee724a998f64980e62a4dd51 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 06:06:52.373807 13908 raft_consensus.cc:729] T ab6fcfff344d4968917c9d5991f20398 P 5124d225ee724a998f64980e62a4dd51 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5124d225ee724a998f64980e62a4dd51, State: Initialized, Role: FOLLOWER
I20260620 06:06:52.374240 13908 consensus_queue.cc:260] T ab6fcfff344d4968917c9d5991f20398 P 5124d225ee724a998f64980e62a4dd51 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5124d225ee724a998f64980e62a4dd51" member_type: VOTER last_known_addr { host: "127.11.102.1" port: 33471 } } peers { permanent_uuid: "73042c6c0e8a4affa6e460bb19d7c5c9" member_type: VOTER last_known_addr { host: "127.11.102.2" port: 43331 } } peers { permanent_uuid: "e61acac2279e47ffbb9af68a1df4d31d" member_type: VOTER last_known_addr { host: "127.11.102.3" port: 38435 } }
I20260620 06:06:52.375125 13908 ts_tablet_manager.cc:1428] T ab6fcfff344d4968917c9d5991f20398 P 5124d225ee724a998f64980e62a4dd51: Time spent starting tablet: real 0.005s user 0.004s sys 0.001s
I20260620 06:06:52.375214 13603 heartbeater.cc:503] Master 127.11.102.62:39839 was elected leader, sending a full tablet report...
I20260620 06:06:52.377490 13907 raft_consensus.cc:348] T ab6fcfff344d4968917c9d5991f20398 P 73042c6c0e8a4affa6e460bb19d7c5c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5124d225ee724a998f64980e62a4dd51" member_type: VOTER last_known_addr { host: "127.11.102.1" port: 33471 } } peers { permanent_uuid: "73042c6c0e8a4affa6e460bb19d7c5c9" member_type: VOTER last_known_addr { host: "127.11.102.2" port: 43331 } } peers { permanent_uuid: "e61acac2279e47ffbb9af68a1df4d31d" member_type: VOTER last_known_addr { host: "127.11.102.3" port: 38435 } }
I20260620 06:06:52.377756 13907 raft_consensus.cc:374] T ab6fcfff344d4968917c9d5991f20398 P 73042c6c0e8a4affa6e460bb19d7c5c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 06:06:52.377820 13907 raft_consensus.cc:729] T ab6fcfff344d4968917c9d5991f20398 P 73042c6c0e8a4affa6e460bb19d7c5c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73042c6c0e8a4affa6e460bb19d7c5c9, State: Initialized, Role: FOLLOWER
I20260620 06:06:52.378194 13907 consensus_queue.cc:260] T ab6fcfff344d4968917c9d5991f20398 P 73042c6c0e8a4affa6e460bb19d7c5c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5124d225ee724a998f64980e62a4dd51" member_type: VOTER last_known_addr { host: "127.11.102.1" port: 33471 } } peers { permanent_uuid: "73042c6c0e8a4affa6e460bb19d7c5c9" member_type: VOTER last_known_addr { host: "127.11.102.2" port: 43331 } } peers { permanent_uuid: "e61acac2279e47ffbb9af68a1df4d31d" member_type: VOTER last_known_addr { host: "127.11.102.3" port: 38435 } }
I20260620 06:06:52.378715 13743 heartbeater.cc:503] Master 127.11.102.62:39839 was elected leader, sending a full tablet report...
I20260620 06:06:52.378965 13907 ts_tablet_manager.cc:1428] T ab6fcfff344d4968917c9d5991f20398 P 73042c6c0e8a4affa6e460bb19d7c5c9: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
W20260620 06:06:52.379205 13604 tablet.cc:2369] T ab6fcfff344d4968917c9d5991f20398 P 5124d225ee724a998f64980e62a4dd51: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 06:06:52.413658 13912 raft_consensus.cc:482] T ab6fcfff344d4968917c9d5991f20398 P e61acac2279e47ffbb9af68a1df4d31d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 06:06:52.413884 13912 raft_consensus.cc:504] T ab6fcfff344d4968917c9d5991f20398 P e61acac2279e47ffbb9af68a1df4d31d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5124d225ee724a998f64980e62a4dd51" member_type: VOTER last_known_addr { host: "127.11.102.1" port: 33471 } } peers { permanent_uuid: "73042c6c0e8a4affa6e460bb19d7c5c9" member_type: VOTER last_known_addr { host: "127.11.102.2" port: 43331 } } peers { permanent_uuid: "e61acac2279e47ffbb9af68a1df4d31d" member_type: VOTER last_known_addr { host: "127.11.102.3" port: 38435 } }
I20260620 06:06:52.415026 13912 leader_election.cc:288] T ab6fcfff344d4968917c9d5991f20398 P e61acac2279e47ffbb9af68a1df4d31d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5124d225ee724a998f64980e62a4dd51 (127.11.102.1:33471), 73042c6c0e8a4affa6e460bb19d7c5c9 (127.11.102.2:43331)
I20260620 06:06:52.419528 13902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:06:52.415534 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.102.3:59227 (local address 127.11.102.1:33471)
0620 06:06:52.415683 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0620 06:06:52.415686 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 06:06:52.415698 (+ 12us) server_negotiation.cc:407] Connection header received
0620 06:06:52.415744 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:06:52.415746 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:06:52.415783 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:06:52.415891 (+ 108us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 06:06:52.416203 (+ 312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:06:52.417014 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:06:52.418396 (+ 1382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:06:52.418978 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:06:52.419103 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:06:52.419176 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0620 06:06:52.419371 (+ 195us) server_negotiation.cc:299] Negotiation successful
0620 06:06:52.419429 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260620 06:06:52.419559 13776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:06:52.415322 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.102.1:33471 (local address 127.11.102.3:59227)
0620 06:06:52.415430 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0620 06:06:52.415446 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 06:06:52.415552 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:06:52.415921 (+ 369us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:06:52.415925 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:06:52.415954 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 06:06:52.416056 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:06:52.416062 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:06:52.417119 (+ 1057us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:06:52.417124 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:06:52.418273 (+ 1149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:06:52.418282 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:06:52.419099 (+ 817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:06:52.419102 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:06:52.419240 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:06:52.419270 (+ 30us) client_negotiation.cc:769] Sending connection context
0620 06:06:52.419350 (+ 80us) client_negotiation.cc:240] Negotiation successful
0620 06:06:52.419423 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30,"mutex_wait_us":39}
I20260620 06:06:52.420150 13557 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab6fcfff344d4968917c9d5991f20398" candidate_uuid: "e61acac2279e47ffbb9af68a1df4d31d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5124d225ee724a998f64980e62a4dd51" is_pre_election: true
I20260620 06:06:52.420372 13903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:06:52.415444 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.102.3:47333 (local address 127.11.102.2:43331)
0620 06:06:52.415554 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0620 06:06:52.415558 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 06:06:52.416138 (+ 580us) server_negotiation.cc:407] Connection header received
0620 06:06:52.416236 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:06:52.416238 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:06:52.416278 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:06:52.416336 (+ 58us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 06:06:52.417060 (+ 724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:06:52.417838 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:06:52.419193 (+ 1355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:06:52.419846 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:06:52.419975 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:06:52.420021 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0620 06:06:52.420201 (+ 180us) server_negotiation.cc:299] Negotiation successful
0620 06:06:52.420267 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260620 06:06:52.420385 13915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:06:52.415322 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.102.2:43331 (local address 127.11.102.3:47333)
0620 06:06:52.416022 (+ 700us) negotiation.cc:107] Waiting for socket to connect
0620 06:06:52.416034 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 06:06:52.416126 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:06:52.416365 (+ 239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:06:52.416368 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:06:52.416386 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 06:06:52.416945 (+ 559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:06:52.416953 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:06:52.417945 (+ 992us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:06:52.417947 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:06:52.419071 (+ 1124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:06:52.419081 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:06:52.419971 (+ 890us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:06:52.419974 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:06:52.420083 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:06:52.420099 (+ 16us) client_negotiation.cc:769] Sending connection context
0620 06:06:52.420187 (+ 88us) client_negotiation.cc:240] Negotiation successful
0620 06:06:52.420248 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":624,"mutex_wait_us":375,"thread_start_us":112,"threads_started":1}
I20260620 06:06:52.420486 13557 raft_consensus.cc:2454] T ab6fcfff344d4968917c9d5991f20398 P 5124d225ee724a998f64980e62a4dd51 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e61acac2279e47ffbb9af68a1df4d31d in term 0.
I20260620 06:06:52.420977 13690 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab6fcfff344d4968917c9d5991f20398" candidate_uuid: "e61acac2279e47ffbb9af68a1df4d31d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73042c6c0e8a4affa6e460bb19d7c5c9" is_pre_election: true
I20260620 06:06:52.421001 13771 leader_election.cc:302] T ab6fcfff344d4968917c9d5991f20398 P e61acac2279e47ffbb9af68a1df4d31d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5124d225ee724a998f64980e62a4dd51, e61acac2279e47ffbb9af68a1df4d31d; no voters:
I20260620 06:06:52.421295 13690 raft_consensus.cc:2454] T ab6fcfff344d4968917c9d5991f20398 P 73042c6c0e8a4affa6e460bb19d7c5c9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e61acac2279e47ffbb9af68a1df4d31d in term 0.
I20260620 06:06:52.421308 13912 raft_consensus.cc:2781] T ab6fcfff344d4968917c9d5991f20398 P e61acac2279e47ffbb9af68a1df4d31d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 06:06:52.421386 13912 raft_consensus.cc:482] T ab6fcfff344d4968917c9d5991f20398 P e61acac2279e47ffbb9af68a1df4d31d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 06:06:52.421429 13912 raft_consensus.cc:3037] T ab6fcfff344d4968917c9d5991f20398 P e61acac2279e47ffbb9af68a1df4d31d [term 0 FOLLOWER]: Advancing to term 1
I20260620 06:06:52.422712 13912 raft_consensus.cc:504] T ab6fcfff344d4968917c9d5991f20398 P e61acac2279e47ffbb9af68a1df4d31d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5124d225ee724a998f64980e62a4dd51" member_type: VOTER last_known_addr { host: "127.11.102.1" port: 33471 } } peers { permanent_uuid: "73042c6c0e8a4affa6e460bb19d7c5c9" member_type: VOTER last_known_addr { host: "127.11.102.2" port: 43331 } } peers { permanent_uuid: "e61acac2279e47ffbb9af68a1df4d31d" member_type: VOTER last_known_addr { host: "127.11.102.3" port: 38435 } }
I20260620 06:06:52.423127 13912 leader_election.cc:288] T ab6fcfff344d4968917c9d5991f20398 P e61acac2279e47ffbb9af68a1df4d31d [CANDIDATE]: Term 1 election: Requested vote from peers 5124d225ee724a998f64980e62a4dd51 (127.11.102.1:33471), 73042c6c0e8a4affa6e460bb19d7c5c9 (127.11.102.2:43331)
I20260620 06:06:52.423434 13690 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab6fcfff344d4968917c9d5991f20398" candidate_uuid: "e61acac2279e47ffbb9af68a1df4d31d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73042c6c0e8a4affa6e460bb19d7c5c9"
I20260620 06:06:52.423453 13557 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab6fcfff344d4968917c9d5991f20398" candidate_uuid: "e61acac2279e47ffbb9af68a1df4d31d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5124d225ee724a998f64980e62a4dd51"
I20260620 06:06:52.423527 13690 raft_consensus.cc:3037] T ab6fcfff344d4968917c9d5991f20398 P 73042c6c0e8a4affa6e460bb19d7c5c9 [term 0 FOLLOWER]: Advancing to term 1
I20260620 06:06:52.423586 13557 raft_consensus.cc:3037] T ab6fcfff344d4968917c9d5991f20398 P 5124d225ee724a998f64980e62a4dd51 [term 0 FOLLOWER]: Advancing to term 1
I20260620 06:06:52.424754 13690 raft_consensus.cc:2454] T ab6fcfff344d4968917c9d5991f20398 P 73042c6c0e8a4affa6e460bb19d7c5c9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e61acac2279e47ffbb9af68a1df4d31d in term 1.
I20260620 06:06:52.424783 13557 raft_consensus.cc:2454] T ab6fcfff344d4968917c9d5991f20398 P 5124d225ee724a998f64980e62a4dd51 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e61acac2279e47ffbb9af68a1df4d31d in term 1.
I20260620 06:06:52.425062 13768 leader_election.cc:302] T ab6fcfff344d4968917c9d5991f20398 P e61acac2279e47ffbb9af68a1df4d31d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 73042c6c0e8a4affa6e460bb19d7c5c9, e61acac2279e47ffbb9af68a1df4d31d; no voters:
I20260620 06:06:52.425226 13912 raft_consensus.cc:2781] T ab6fcfff344d4968917c9d5991f20398 P e61acac2279e47ffbb9af68a1df4d31d [term 1 FOLLOWER]: Leader election won for term 1
I20260620 06:06:52.425505 13912 raft_consensus.cc:686] T ab6fcfff344d4968917c9d5991f20398 P e61acac2279e47ffbb9af68a1df4d31d [term 1 LEADER]: Becoming Leader. State: Replica: e61acac2279e47ffbb9af68a1df4d31d, State: Running, Role: LEADER
I20260620 06:06:52.425828 13912 consensus_queue.cc:237] T ab6fcfff344d4968917c9d5991f20398 P e61acac2279e47ffbb9af68a1df4d31d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5124d225ee724a998f64980e62a4dd51" member_type: VOTER last_known_addr { host: "127.11.102.1" port: 33471 } } peers { permanent_uuid: "73042c6c0e8a4affa6e460bb19d7c5c9" member_type: VOTER last_known_addr { host: "127.11.102.2" port: 43331 } } peers { permanent_uuid: "e61acac2279e47ffbb9af68a1df4d31d" member_type: VOTER last_known_addr { host: "127.11.102.3" port: 38435 } }
I20260620 06:06:52.429051 13400 catalog_manager.cc:5495] T ab6fcfff344d4968917c9d5991f20398 P e61acac2279e47ffbb9af68a1df4d31d reported cstate change: term changed from 0 to 1, leader changed from <none> to e61acac2279e47ffbb9af68a1df4d31d (127.11.102.3). New cstate: current_term: 1 leader_uuid: "e61acac2279e47ffbb9af68a1df4d31d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5124d225ee724a998f64980e62a4dd51" member_type: VOTER last_known_addr { host: "127.11.102.1" port: 33471 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "73042c6c0e8a4affa6e460bb19d7c5c9" member_type: VOTER last_known_addr { host: "127.11.102.2" port: 43331 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e61acac2279e47ffbb9af68a1df4d31d" member_type: VOTER last_known_addr { host: "127.11.102.3" port: 38435 } health_report { overall_health: HEALTHY } } }
Jun 20 06:06:52 dist-test-slave-9hm5 krb5kdc[13355](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935611, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.102.62@KRBTEST.COM for kudu/127.11.102.62@KRBTEST.COM
I20260620 06:06:52.485771 13923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:06:52.477386 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.102.62:39839 (local address 127.0.0.1:57828)
0620 06:06:52.477704 (+ 318us) negotiation.cc:107] Waiting for socket to connect
0620 06:06:52.477717 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 06:06:52.477850 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:06:52.478109 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:06:52.478112 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:06:52.478552 (+ 440us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:06:52.478642 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:06:52.478650 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:06:52.479387 (+ 737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:06:52.479393 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:06:52.479970 (+ 577us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:06:52.479978 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:06:52.480396 (+ 418us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:06:52.480399 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:06:52.480471 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:06:52.481025 (+ 554us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:06:52.481036 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:06:52.482473 (+ 1437us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:06:52.484319 (+ 1846us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:06:52.484327 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:06:52.484333 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:06:52.484573 (+ 240us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:06:52.484816 (+ 243us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:06:52.484818 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:06:52.484819 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:06:52.484895 (+ 76us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:06:52.485299 (+ 404us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:06:52.485305 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:06:52.485407 (+ 102us) client_negotiation.cc:769] Sending connection context
0620 06:06:52.485500 (+ 93us) client_negotiation.cc:240] Negotiation successful
0620 06:06:52.485610 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":231,"thread_start_us":102,"threads_started":1}
I20260620 06:06:52.485792 13449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:06:52.477484 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57828 (local address 127.11.102.62:39839)
0620 06:06:52.477634 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0620 06:06:52.477639 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 06:06:52.477798 (+ 159us) server_negotiation.cc:407] Connection header received
0620 06:06:52.477973 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:06:52.477976 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:06:52.478010 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:06:52.478092 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:06:52.478763 (+ 671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:06:52.479283 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:06:52.480085 (+ 802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:06:52.480292 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:06:52.480396 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:06:52.482609 (+ 2213us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:06:52.482612 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:06:52.482614 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:06:52.482636 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:06:52.484214 (+ 1578us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:06:52.484663 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:06:52.484666 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:06:52.484668 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:06:52.484737 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:06:52.485004 (+ 267us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:06:52.485007 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:06:52.485008 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:06:52.485194 (+ 186us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:06:52.485281 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0620 06:06:52.485567 (+ 286us) server_negotiation.cc:299] Negotiation successful
0620 06:06:52.485662 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260620 06:06:52.490103 13400 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.11.102.62@KRBTEST.COM'} at 127.0.0.1:57828:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260620 06:06:52.490844 13400 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 06:06:52.495711 13677 tablet_service.cc:1467] Processing CreateTablet for tablet 04d800ddb0a04b3db00030c30e1bb34c (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=484281e44a91417b9379ea6ecae50e70]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 06:06:52.495508 13537 tablet_service.cc:1467] Processing CreateTablet for tablet 04d800ddb0a04b3db00030c30e1bb34c (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=484281e44a91417b9379ea6ecae50e70]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 06:06:52.495561 13817 tablet_service.cc:1467] Processing CreateTablet for tablet 04d800ddb0a04b3db00030c30e1bb34c (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=484281e44a91417b9379ea6ecae50e70]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 06:06:52.496309 13817 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 04d800ddb0a04b3db00030c30e1bb34c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 06:06:52.496309 13537 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 04d800ddb0a04b3db00030c30e1bb34c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 06:06:52.496309 13677 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 04d800ddb0a04b3db00030c30e1bb34c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 06:06:52.498814 13907 tablet_bootstrap.cc:492] T 04d800ddb0a04b3db00030c30e1bb34c P 73042c6c0e8a4affa6e460bb19d7c5c9: Bootstrap starting.
I20260620 06:06:52.499125 13908 tablet_bootstrap.cc:492] T 04d800ddb0a04b3db00030c30e1bb34c P 5124d225ee724a998f64980e62a4dd51: Bootstrap starting.
I20260620 06:06:52.499148 13906 tablet_bootstrap.cc:492] T 04d800ddb0a04b3db00030c30e1bb34c P e61acac2279e47ffbb9af68a1df4d31d: Bootstrap starting.
I20260620 06:06:52.499909 13907 tablet_bootstrap.cc:654] T 04d800ddb0a04b3db00030c30e1bb34c P 73042c6c0e8a4affa6e460bb19d7c5c9: Neither blocks nor log segments found. Creating new log.
I20260620 06:06:52.499981 13908 tablet_bootstrap.cc:654] T 04d800ddb0a04b3db00030c30e1bb34c P 5124d225ee724a998f64980e62a4dd51: Neither blocks nor log segments found. Creating new log.
I20260620 06:06:52.500144 13906 tablet_bootstrap.cc:654] T 04d800ddb0a04b3db00030c30e1bb34c P e61acac2279e47ffbb9af68a1df4d31d: Neither blocks nor log segments found. Creating new log.
I20260620 06:06:52.501180 13908 tablet_bootstrap.cc:492] T 04d800ddb0a04b3db00030c30e1bb34c P 5124d225ee724a998f64980e62a4dd51: No bootstrap required, opened a new log
I20260620 06:06:52.501264 13908 ts_tablet_manager.cc:1397] T 04d800ddb0a04b3db00030c30e1bb34c P 5124d225ee724a998f64980e62a4dd51: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260620 06:06:52.501380 13906 tablet_bootstrap.cc:492] T 04d800ddb0a04b3db00030c30e1bb34c P e61acac2279e47ffbb9af68a1df4d31d: No bootstrap required, opened a new log
I20260620 06:06:52.501452 13906 ts_tablet_manager.cc:1397] T 04d800ddb0a04b3db00030c30e1bb34c P e61acac2279e47ffbb9af68a1df4d31d: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260620 06:06:52.501605 13907 tablet_bootstrap.cc:492] T 04d800ddb0a04b3db00030c30e1bb34c P 73042c6c0e8a4affa6e460bb19d7c5c9: No bootstrap required, opened a new log
I20260620 06:06:52.501624 13908 raft_consensus.cc:348] T 04d800ddb0a04b3db00030c30e1bb34c P 5124d225ee724a998f64980e62a4dd51 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5124d225ee724a998f64980e62a4dd51" member_type: VOTER last_known_addr { host: "127.11.102.1" port: 33471 } } peers { permanent_uuid: "73042c6c0e8a4affa6e460bb19d7c5c9" member_type: VOTER last_known_addr { host: "127.11.102.2" port: 43331 } } peers { permanent_uuid: "e61acac2279e47ffbb9af68a1df4d31d" member_type: VOTER last_known_addr { host: "127.11.102.3" port: 38435 } }
I20260620 06:06:52.501673 13907 ts_tablet_manager.cc:1397] T 04d800ddb0a04b3db00030c30e1bb34c P 73042c6c0e8a4affa6e460bb19d7c5c9: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260620 06:06:52.501703 13908 raft_consensus.cc:374] T 04d800ddb0a04b3db00030c30e1bb34c P 5124d225ee724a998f64980e62a4dd51 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 06:06:52.501725 13908 raft_consensus.cc:729] T 04d800ddb0a04b3db00030c30e1bb34c P 5124d225ee724a998f64980e62a4dd51 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5124d225ee724a998f64980e62a4dd51, State: Initialized, Role: FOLLOWER
I20260620 06:06:52.501853 13908 consensus_queue.cc:260] T 04d800ddb0a04b3db00030c30e1bb34c P 5124d225ee724a998f64980e62a4dd51 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5124d225ee724a998f64980e62a4dd51" member_type: VOTER last_known_addr { host: "127.11.102.1" port: 33471 } } peers { permanent_uuid: "73042c6c0e8a4affa6e460bb19d7c5c9" member_type: VOTER last_known_addr { host: "127.11.102.2" port: 43331 } } peers { permanent_uuid: "e61acac2279e47ffbb9af68a1df4d31d" member_type: VOTER last_known_addr { host: "127.11.102.3" port: 38435 } }
I20260620 06:06:52.501900 13906 raft_consensus.cc:348] T 04d800ddb0a04b3db00030c30e1bb34c P e61acac2279e47ffbb9af68a1df4d31d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5124d225ee724a998f64980e62a4dd51" member_type: VOTER last_known_addr { host: "127.11.102.1" port: 33471 } } peers { permanent_uuid: "73042c6c0e8a4affa6e460bb19d7c5c9" member_type: VOTER last_known_addr { host: "127.11.102.2" port: 43331 } } peers { permanent_uuid: "e61acac2279e47ffbb9af68a1df4d31d" member_type: VOTER last_known_addr { host: "127.11.102.3" port: 38435 } }
I20260620 06:06:52.502000 13906 raft_consensus.cc:374] T 04d800ddb0a04b3db00030c30e1bb34c P e61acac2279e47ffbb9af68a1df4d31d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 06:06:52.502031 13906 raft_consensus.cc:729] T 04d800ddb0a04b3db00030c30e1bb34c P e61acac2279e47ffbb9af68a1df4d31d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e61acac2279e47ffbb9af68a1df4d31d, State: Initialized, Role: FOLLOWER
I20260620 06:06:52.502135 13908 ts_tablet_manager.cc:1428] T 04d800ddb0a04b3db00030c30e1bb34c P 5124d225ee724a998f64980e62a4dd51: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260620 06:06:52.502090 13907 raft_consensus.cc:348] T 04d800ddb0a04b3db00030c30e1bb34c P 73042c6c0e8a4affa6e460bb19d7c5c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5124d225ee724a998f64980e62a4dd51" member_type: VOTER last_known_addr { host: "127.11.102.1" port: 33471 } } peers { permanent_uuid: "73042c6c0e8a4affa6e460bb19d7c5c9" member_type: VOTER last_known_addr { host: "127.11.102.2" port: 43331 } } peers { permanent_uuid: "e61acac2279e47ffbb9af68a1df4d31d" member_type: VOTER last_known_addr { host: "127.11.102.3" port: 38435 } }
I20260620 06:06:52.502105 13913 tablet_replica.cc:439] T 04d800ddb0a04b3db00030c30e1bb34c P 5124d225ee724a998f64980e62a4dd51: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5124d225ee724a998f64980e62a4dd51" member_type: VOTER last_known_addr { host: "127.11.102.1" port: 33471 } } peers { permanent_uuid: "73042c6c0e8a4affa6e460bb19d7c5c9" member_type: VOTER last_known_addr { host: "127.11.102.2" port: 43331 } } peers { permanent_uuid: "e61acac2279e47ffbb9af68a1df4d31d" member_type: VOTER last_known_addr { host: "127.11.102.3" port: 38435 } } }
I20260620 06:06:52.502188 13907 raft_consensus.cc:374] T 04d800ddb0a04b3db00030c30e1bb34c P 73042c6c0e8a4affa6e460bb19d7c5c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 06:06:52.502151 13906 consensus_queue.cc:260] T 04d800ddb0a04b3db00030c30e1bb34c P e61acac2279e47ffbb9af68a1df4d31d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5124d225ee724a998f64980e62a4dd51" member_type: VOTER last_known_addr { host: "127.11.102.1" port: 33471 } } peers { permanent_uuid: "73042c6c0e8a4affa6e460bb19d7c5c9" member_type: VOTER last_known_addr { host: "127.11.102.2" port: 43331 } } peers { permanent_uuid: "e61acac2279e47ffbb9af68a1df4d31d" member_type: VOTER last_known_addr { host: "127.11.102.3" port: 38435 } }
I20260620 06:06:52.502218 13907 raft_consensus.cc:729] T 04d800ddb0a04b3db00030c30e1bb34c P 73042c6c0e8a4affa6e460bb19d7c5c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73042c6c0e8a4affa6e460bb19d7c5c9, State: Initialized, Role: FOLLOWER
I20260620 06:06:52.502226 13913 tablet_replica.cc:442] T 04d800ddb0a04b3db00030c30e1bb34c P 5124d225ee724a998f64980e62a4dd51: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 06:06:52.502331 13907 consensus_queue.cc:260] T 04d800ddb0a04b3db00030c30e1bb34c P 73042c6c0e8a4affa6e460bb19d7c5c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5124d225ee724a998f64980e62a4dd51" member_type: VOTER last_known_addr { host: "127.11.102.1" port: 33471 } } peers { permanent_uuid: "73042c6c0e8a4affa6e460bb19d7c5c9" member_type: VOTER last_known_addr { host: "127.11.102.2" port: 43331 } } peers { permanent_uuid: "e61acac2279e47ffbb9af68a1df4d31d" member_type: VOTER last_known_addr { host: "127.11.102.3" port: 38435 } }
I20260620 06:06:52.502393 13912 tablet_replica.cc:439] T 04d800ddb0a04b3db00030c30e1bb34c P e61acac2279e47ffbb9af68a1df4d31d: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5124d225ee724a998f64980e62a4dd51" member_type: VOTER last_known_addr { host: "127.11.102.1" port: 33471 } } peers { permanent_uuid: "73042c6c0e8a4affa6e460bb19d7c5c9" member_type: VOTER last_known_addr { host: "127.11.102.2" port: 43331 } } peers { permanent_uuid: "e61acac2279e47ffbb9af68a1df4d31d" member_type: VOTER last_known_addr { host: "127.11.102.3" port: 38435 } } }
I20260620 06:06:52.502511 13912 tablet_replica.cc:442] T 04d800ddb0a04b3db00030c30e1bb34c P e61acac2279e47ffbb9af68a1df4d31d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 06:06:52.502573 13914 tablet_replica.cc:439] T 04d800ddb0a04b3db00030c30e1bb34c P 73042c6c0e8a4affa6e460bb19d7c5c9: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5124d225ee724a998f64980e62a4dd51" member_type: VOTER last_known_addr { host: "127.11.102.1" port: 33471 } } peers { permanent_uuid: "73042c6c0e8a4affa6e460bb19d7c5c9" member_type: VOTER last_known_addr { host: "127.11.102.2" port: 43331 } } peers { permanent_uuid: "e61acac2279e47ffbb9af68a1df4d31d" member_type: VOTER last_known_addr { host: "127.11.102.3" port: 38435 } } }
I20260620 06:06:52.502732 13914 tablet_replica.cc:442] T 04d800ddb0a04b3db00030c30e1bb34c P 73042c6c0e8a4affa6e460bb19d7c5c9: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 06:06:52.502583 13907 ts_tablet_manager.cc:1428] T 04d800ddb0a04b3db00030c30e1bb34c P 73042c6c0e8a4affa6e460bb19d7c5c9: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260620 06:06:52.502410 13906 ts_tablet_manager.cc:1428] T 04d800ddb0a04b3db00030c30e1bb34c P e61acac2279e47ffbb9af68a1df4d31d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
W20260620 06:06:52.543756 13884 tablet.cc:2369] T 04d800ddb0a04b3db00030c30e1bb34c P e61acac2279e47ffbb9af68a1df4d31d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 06:06:52.544291 13914 raft_consensus.cc:482] T 04d800ddb0a04b3db00030c30e1bb34c P 73042c6c0e8a4affa6e460bb19d7c5c9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 06:06:52.544405 13914 raft_consensus.cc:504] T 04d800ddb0a04b3db00030c30e1bb34c P 73042c6c0e8a4affa6e460bb19d7c5c9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5124d225ee724a998f64980e62a4dd51" member_type: VOTER last_known_addr { host: "127.11.102.1" port: 33471 } } peers { permanent_uuid: "73042c6c0e8a4affa6e460bb19d7c5c9" member_type: VOTER last_known_addr { host: "127.11.102.2" port: 43331 } } peers { permanent_uuid: "e61acac2279e47ffbb9af68a1df4d31d" member_type: VOTER last_known_addr { host: "127.11.102.3" port: 38435 } }
I20260620 06:06:52.545534 13914 leader_election.cc:288] T 04d800ddb0a04b3db00030c30e1bb34c P 73042c6c0e8a4affa6e460bb19d7c5c9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5124d225ee724a998f64980e62a4dd51 (127.11.102.1:33471), e61acac2279e47ffbb9af68a1df4d31d (127.11.102.3:38435)
I20260620 06:06:52.550211 13902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:06:52.545822 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.102.2:43623 (local address 127.11.102.1:33471)
0620 06:06:52.546017 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0620 06:06:52.546022 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 06:06:52.546037 (+ 15us) server_negotiation.cc:407] Connection header received
0620 06:06:52.546157 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:06:52.546160 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:06:52.546202 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:06:52.546270 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 06:06:52.546584 (+ 314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:06:52.547375 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:06:52.549097 (+ 1722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:06:52.549652 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:06:52.549788 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:06:52.549841 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0620 06:06:52.550014 (+ 173us) server_negotiation.cc:299] Negotiation successful
0620 06:06:52.550085 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260620 06:06:52.550209 13636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:06:52.545730 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.102.1:33471 (local address 127.11.102.2:43623)
0620 06:06:52.545919 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0620 06:06:52.545936 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 06:06:52.546044 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:06:52.546311 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:06:52.546317 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:06:52.546341 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 06:06:52.546472 (+ 131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:06:52.546479 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:06:52.547468 (+ 989us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:06:52.547470 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:06:52.548945 (+ 1475us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:06:52.548956 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:06:52.549777 (+ 821us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:06:52.549780 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:06:52.549885 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:06:52.549913 (+ 28us) client_negotiation.cc:769] Sending connection context
0620 06:06:52.549999 (+ 86us) client_negotiation.cc:240] Negotiation successful
0620 06:06:52.550069 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":104,"mutex_wait_us":61}
I20260620 06:06:52.550328 13905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:06:52.545843 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.102.2:49037 (local address 127.11.102.3:38435)
0620 06:06:52.545983 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0620 06:06:52.545988 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 06:06:52.546355 (+ 367us) server_negotiation.cc:407] Connection header received
0620 06:06:52.546515 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:06:52.546518 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:06:52.546588 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:06:52.546689 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 06:06:52.546928 (+ 239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:06:52.547661 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:06:52.549192 (+ 1531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:06:52.549727 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:06:52.549861 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:06:52.549908 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0620 06:06:52.550129 (+ 221us) server_negotiation.cc:299] Negotiation successful
0620 06:06:52.550200 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260620 06:06:52.550570 13927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:06:52.545750 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.102.3:38435 (local address 127.11.102.2:49037)
0620 06:06:52.546236 (+ 486us) negotiation.cc:107] Waiting for socket to connect
0620 06:06:52.546291 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0620 06:06:52.546407 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:06:52.546708 (+ 301us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:06:52.546710 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:06:52.546733 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 06:06:52.546824 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:06:52.546828 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:06:52.547761 (+ 933us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:06:52.547765 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:06:52.549001 (+ 1236us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:06:52.549010 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:06:52.549867 (+ 857us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:06:52.549871 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:06:52.549991 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:06:52.550015 (+ 24us) client_negotiation.cc:769] Sending connection context
0620 06:06:52.550112 (+ 97us) client_negotiation.cc:240] Negotiation successful
0620 06:06:52.550180 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":431,"mutex_wait_us":13,"thread_start_us":156,"threads_started":1}
I20260620 06:06:52.550814 13557 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04d800ddb0a04b3db00030c30e1bb34c" candidate_uuid: "73042c6c0e8a4affa6e460bb19d7c5c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5124d225ee724a998f64980e62a4dd51" is_pre_election: true
I20260620 06:06:52.550976 13557 raft_consensus.cc:2454] T 04d800ddb0a04b3db00030c30e1bb34c P 5124d225ee724a998f64980e62a4dd51 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 73042c6c0e8a4affa6e460bb19d7c5c9 in term 0.
I20260620 06:06:52.551209 13837 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04d800ddb0a04b3db00030c30e1bb34c" candidate_uuid: "73042c6c0e8a4affa6e460bb19d7c5c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e61acac2279e47ffbb9af68a1df4d31d" is_pre_election: true
I20260620 06:06:52.551352 13631 leader_election.cc:302] T 04d800ddb0a04b3db00030c30e1bb34c P 73042c6c0e8a4affa6e460bb19d7c5c9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5124d225ee724a998f64980e62a4dd51, 73042c6c0e8a4affa6e460bb19d7c5c9; no voters:
I20260620 06:06:52.551424 13837 raft_consensus.cc:2454] T 04d800ddb0a04b3db00030c30e1bb34c P e61acac2279e47ffbb9af68a1df4d31d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 73042c6c0e8a4affa6e460bb19d7c5c9 in term 0.
I20260620 06:06:52.551764 13914 raft_consensus.cc:2781] T 04d800ddb0a04b3db00030c30e1bb34c P 73042c6c0e8a4affa6e460bb19d7c5c9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 06:06:52.551831 13914 raft_consensus.cc:482] T 04d800ddb0a04b3db00030c30e1bb34c P 73042c6c0e8a4affa6e460bb19d7c5c9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 06:06:52.551883 13914 raft_consensus.cc:3037] T 04d800ddb0a04b3db00030c30e1bb34c P 73042c6c0e8a4affa6e460bb19d7c5c9 [term 0 FOLLOWER]: Advancing to term 1
I20260620 06:06:52.552911 13914 raft_consensus.cc:504] T 04d800ddb0a04b3db00030c30e1bb34c P 73042c6c0e8a4affa6e460bb19d7c5c9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5124d225ee724a998f64980e62a4dd51" member_type: VOTER last_known_addr { host: "127.11.102.1" port: 33471 } } peers { permanent_uuid: "73042c6c0e8a4affa6e460bb19d7c5c9" member_type: VOTER last_known_addr { host: "127.11.102.2" port: 43331 } } peers { permanent_uuid: "e61acac2279e47ffbb9af68a1df4d31d" member_type: VOTER last_known_addr { host: "127.11.102.3" port: 38435 } }
I20260620 06:06:52.553241 13914 leader_election.cc:288] T 04d800ddb0a04b3db00030c30e1bb34c P 73042c6c0e8a4affa6e460bb19d7c5c9 [CANDIDATE]: Term 1 election: Requested vote from peers 5124d225ee724a998f64980e62a4dd51 (127.11.102.1:33471), e61acac2279e47ffbb9af68a1df4d31d (127.11.102.3:38435)
I20260620 06:06:52.553578 13837 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04d800ddb0a04b3db00030c30e1bb34c" candidate_uuid: "73042c6c0e8a4affa6e460bb19d7c5c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e61acac2279e47ffbb9af68a1df4d31d"
I20260620 06:06:52.553579 13557 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04d800ddb0a04b3db00030c30e1bb34c" candidate_uuid: "73042c6c0e8a4affa6e460bb19d7c5c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5124d225ee724a998f64980e62a4dd51"
I20260620 06:06:52.553689 13837 raft_consensus.cc:3037] T 04d800ddb0a04b3db00030c30e1bb34c P e61acac2279e47ffbb9af68a1df4d31d [term 0 FOLLOWER]: Advancing to term 1
I20260620 06:06:52.553695 13557 raft_consensus.cc:3037] T 04d800ddb0a04b3db00030c30e1bb34c P 5124d225ee724a998f64980e62a4dd51 [term 0 FOLLOWER]: Advancing to term 1
I20260620 06:06:52.554788 13557 raft_consensus.cc:2454] T 04d800ddb0a04b3db00030c30e1bb34c P 5124d225ee724a998f64980e62a4dd51 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 73042c6c0e8a4affa6e460bb19d7c5c9 in term 1.
I20260620 06:06:52.554788 13837 raft_consensus.cc:2454] T 04d800ddb0a04b3db00030c30e1bb34c P e61acac2279e47ffbb9af68a1df4d31d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 73042c6c0e8a4affa6e460bb19d7c5c9 in term 1.
I20260620 06:06:52.555047 13631 leader_election.cc:302] T 04d800ddb0a04b3db00030c30e1bb34c P 73042c6c0e8a4affa6e460bb19d7c5c9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5124d225ee724a998f64980e62a4dd51, 73042c6c0e8a4affa6e460bb19d7c5c9; no voters:
I20260620 06:06:52.555209 13914 raft_consensus.cc:2781] T 04d800ddb0a04b3db00030c30e1bb34c P 73042c6c0e8a4affa6e460bb19d7c5c9 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 06:06:52.555421 13914 raft_consensus.cc:686] T 04d800ddb0a04b3db00030c30e1bb34c P 73042c6c0e8a4affa6e460bb19d7c5c9 [term 1 LEADER]: Becoming Leader. State: Replica: 73042c6c0e8a4affa6e460bb19d7c5c9, State: Running, Role: LEADER
I20260620 06:06:52.555791 13914 consensus_queue.cc:237] T 04d800ddb0a04b3db00030c30e1bb34c P 73042c6c0e8a4affa6e460bb19d7c5c9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5124d225ee724a998f64980e62a4dd51" member_type: VOTER last_known_addr { host: "127.11.102.1" port: 33471 } } peers { permanent_uuid: "73042c6c0e8a4affa6e460bb19d7c5c9" member_type: VOTER last_known_addr { host: "127.11.102.2" port: 43331 } } peers { permanent_uuid: "e61acac2279e47ffbb9af68a1df4d31d" member_type: VOTER last_known_addr { host: "127.11.102.3" port: 38435 } }
I20260620 06:06:52.558079 13928 tablet_replica.cc:439] T 04d800ddb0a04b3db00030c30e1bb34c P 73042c6c0e8a4affa6e460bb19d7c5c9: TxnStatusTablet state changed. Reason: New leader 73042c6c0e8a4affa6e460bb19d7c5c9. Latest consensus state: current_term: 1 leader_uuid: "73042c6c0e8a4affa6e460bb19d7c5c9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5124d225ee724a998f64980e62a4dd51" member_type: VOTER last_known_addr { host: "127.11.102.1" port: 33471 } } peers { permanent_uuid: "73042c6c0e8a4affa6e460bb19d7c5c9" member_type: VOTER last_known_addr { host: "127.11.102.2" port: 43331 } } peers { permanent_uuid: "e61acac2279e47ffbb9af68a1df4d31d" member_type: VOTER last_known_addr { host: "127.11.102.3" port: 38435 } } }
I20260620 06:06:52.558182 13928 tablet_replica.cc:442] T 04d800ddb0a04b3db00030c30e1bb34c P 73042c6c0e8a4affa6e460bb19d7c5c9: This TxnStatusTablet replica's current role is: LEADER
I20260620 06:06:52.558422 13930 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260620 06:06:52.558640 13930 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260620 06:06:52.558957 13399 catalog_manager.cc:5495] T 04d800ddb0a04b3db00030c30e1bb34c P 73042c6c0e8a4affa6e460bb19d7c5c9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 73042c6c0e8a4affa6e460bb19d7c5c9 (127.11.102.2). New cstate: current_term: 1 leader_uuid: "73042c6c0e8a4affa6e460bb19d7c5c9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5124d225ee724a998f64980e62a4dd51" member_type: VOTER last_known_addr { host: "127.11.102.1" port: 33471 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "73042c6c0e8a4affa6e460bb19d7c5c9" member_type: VOTER last_known_addr { host: "127.11.102.2" port: 43331 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e61acac2279e47ffbb9af68a1df4d31d" member_type: VOTER last_known_addr { host: "127.11.102.3" port: 38435 } health_report { overall_health: UNKNOWN } } }
I20260620 06:06:52.571472 13903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:06:52.567903 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50166 (local address 127.11.102.2:43331)
0620 06:06:52.568337 (+ 434us) server_negotiation.cc:206] Beginning negotiation
0620 06:06:52.568340 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 06:06:52.568350 (+ 10us) server_negotiation.cc:407] Connection header received
0620 06:06:52.568397 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:06:52.568399 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:06:52.568442 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:06:52.568541 (+ 99us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 06:06:52.568782 (+ 241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:06:52.569286 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:06:52.570109 (+ 823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:06:52.570336 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:06:52.570443 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:06:52.570726 (+ 283us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 06:06:52.571099 (+ 373us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 06:06:52.571227 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0620 06:06:52.571293 (+ 66us) server_negotiation.cc:299] Negotiation successful
0620 06:06:52.571340 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":349}
I20260620 06:06:52.571480 13923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:06:52.567850 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.102.2:43331 (local address 127.0.0.1:50166)
0620 06:06:52.567990 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0620 06:06:52.568006 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 06:06:52.568104 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:06:52.568552 (+ 448us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:06:52.568555 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:06:52.568562 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 06:06:52.568678 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:06:52.568684 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:06:52.569393 (+ 709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:06:52.569397 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:06:52.569993 (+ 596us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:06:52.570000 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:06:52.570442 (+ 442us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:06:52.570446 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:06:52.570541 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:06:52.570581 (+ 40us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 06:06:52.571202 (+ 621us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 06:06:52.571217 (+ 15us) client_negotiation.cc:769] Sending connection context
0620 06:06:52.571285 (+ 68us) client_negotiation.cc:240] Negotiation successful
0620 06:06:52.571364 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":29}
I20260620 06:06:52.576907 13837 raft_consensus.cc:1261] T 04d800ddb0a04b3db00030c30e1bb34c P e61acac2279e47ffbb9af68a1df4d31d [term 1 FOLLOWER]: Refusing update from remote peer 73042c6c0e8a4affa6e460bb19d7c5c9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 06:06:52.576969 13557 raft_consensus.cc:1261] T 04d800ddb0a04b3db00030c30e1bb34c P 5124d225ee724a998f64980e62a4dd51 [term 1 FOLLOWER]: Refusing update from remote peer 73042c6c0e8a4affa6e460bb19d7c5c9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 06:06:52.577538 13914 consensus_queue.cc:1035] T 04d800ddb0a04b3db00030c30e1bb34c P 73042c6c0e8a4affa6e460bb19d7c5c9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e61acac2279e47ffbb9af68a1df4d31d" member_type: VOTER last_known_addr { host: "127.11.102.3" port: 38435 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 06:06:52.577692 13928 consensus_queue.cc:1035] T 04d800ddb0a04b3db00030c30e1bb34c P 73042c6c0e8a4affa6e460bb19d7c5c9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5124d225ee724a998f64980e62a4dd51" member_type: VOTER last_known_addr { host: "127.11.102.1" port: 33471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 06:06:52.580974 13912 tablet_replica.cc:439] T 04d800ddb0a04b3db00030c30e1bb34c P e61acac2279e47ffbb9af68a1df4d31d: TxnStatusTablet state changed. Reason: New leader 73042c6c0e8a4affa6e460bb19d7c5c9. Latest consensus state: current_term: 1 leader_uuid: "73042c6c0e8a4affa6e460bb19d7c5c9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5124d225ee724a998f64980e62a4dd51" member_type: VOTER last_known_addr { host: "127.11.102.1" port: 33471 } } peers { permanent_uuid: "73042c6c0e8a4affa6e460bb19d7c5c9" member_type: VOTER last_known_addr { host: "127.11.102.2" port: 43331 } } peers { permanent_uuid: "e61acac2279e47ffbb9af68a1df4d31d" member_type: VOTER last_known_addr { host: "127.11.102.3" port: 38435 } } }
I20260620 06:06:52.581103 13912 tablet_replica.cc:442] T 04d800ddb0a04b3db00030c30e1bb34c P e61acac2279e47ffbb9af68a1df4d31d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 06:06:52.581089 13913 tablet_replica.cc:439] T 04d800ddb0a04b3db00030c30e1bb34c P 5124d225ee724a998f64980e62a4dd51: TxnStatusTablet state changed. Reason: New leader 73042c6c0e8a4affa6e460bb19d7c5c9. Latest consensus state: current_term: 1 leader_uuid: "73042c6c0e8a4affa6e460bb19d7c5c9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5124d225ee724a998f64980e62a4dd51" member_type: VOTER last_known_addr { host: "127.11.102.1" port: 33471 } } peers { permanent_uuid: "73042c6c0e8a4affa6e460bb19d7c5c9" member_type: VOTER last_known_addr { host: "127.11.102.2" port: 43331 } } peers { permanent_uuid: "e61acac2279e47ffbb9af68a1df4d31d" member_type: VOTER last_known_addr { host: "127.11.102.3" port: 38435 } } }
I20260620 06:06:52.581197 13913 tablet_replica.cc:442] T 04d800ddb0a04b3db00030c30e1bb34c P 5124d225ee724a998f64980e62a4dd51: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 06:06:52.584771 13912 tablet_replica.cc:439] T 04d800ddb0a04b3db00030c30e1bb34c P e61acac2279e47ffbb9af68a1df4d31d: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "73042c6c0e8a4affa6e460bb19d7c5c9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5124d225ee724a998f64980e62a4dd51" member_type: VOTER last_known_addr { host: "127.11.102.1" port: 33471 } } peers { permanent_uuid: "73042c6c0e8a4affa6e460bb19d7c5c9" member_type: VOTER last_known_addr { host: "127.11.102.2" port: 43331 } } peers { permanent_uuid: "e61acac2279e47ffbb9af68a1df4d31d" member_type: VOTER last_known_addr { host: "127.11.102.3" port: 38435 } } }
I20260620 06:06:52.584885 13912 tablet_replica.cc:442] T 04d800ddb0a04b3db00030c30e1bb34c P e61acac2279e47ffbb9af68a1df4d31d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 06:06:52.585784 13914 tablet_replica.cc:439] T 04d800ddb0a04b3db00030c30e1bb34c P 73042c6c0e8a4affa6e460bb19d7c5c9: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "73042c6c0e8a4affa6e460bb19d7c5c9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5124d225ee724a998f64980e62a4dd51" member_type: VOTER last_known_addr { host: "127.11.102.1" port: 33471 } } peers { permanent_uuid: "73042c6c0e8a4affa6e460bb19d7c5c9" member_type: VOTER last_known_addr { host: "127.11.102.2" port: 43331 } } peers { permanent_uuid: "e61acac2279e47ffbb9af68a1df4d31d" member_type: VOTER last_known_addr { host: "127.11.102.3" port: 38435 } } }
I20260620 06:06:52.585903 13914 tablet_replica.cc:442] T 04d800ddb0a04b3db00030c30e1bb34c P 73042c6c0e8a4affa6e460bb19d7c5c9: This TxnStatusTablet replica's current role is: LEADER
I20260620 06:06:52.585904 13913 tablet_replica.cc:439] T 04d800ddb0a04b3db00030c30e1bb34c P 5124d225ee724a998f64980e62a4dd51: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "73042c6c0e8a4affa6e460bb19d7c5c9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5124d225ee724a998f64980e62a4dd51" member_type: VOTER last_known_addr { host: "127.11.102.1" port: 33471 } } peers { permanent_uuid: "73042c6c0e8a4affa6e460bb19d7c5c9" member_type: VOTER last_known_addr { host: "127.11.102.2" port: 43331 } } peers { permanent_uuid: "e61acac2279e47ffbb9af68a1df4d31d" member_type: VOTER last_known_addr { host: "127.11.102.3" port: 38435 } } }
I20260620 06:06:52.586004 13913 tablet_replica.cc:442] T 04d800ddb0a04b3db00030c30e1bb34c P 5124d225ee724a998f64980e62a4dd51: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 06:06:52.587041 13935 mvcc.cc:205] Tried to move back new op lower bound from 7298808269108133888 to 7298808269032337408. Current Snapshot: MvccSnapshot[applied={T|T < 7298808269108133888}]
I20260620 06:06:52.588071 13934 mvcc.cc:205] Tried to move back new op lower bound from 7298808269108133888 to 7298808269032337408. Current Snapshot: MvccSnapshot[applied={T|T < 7298808269108133888}]
W20260620 06:06:52.588177 13744 tablet.cc:2369] T ab6fcfff344d4968917c9d5991f20398 P 73042c6c0e8a4affa6e460bb19d7c5c9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 06:06:52.588243 13938 tablet_replica.cc:439] T 04d800ddb0a04b3db00030c30e1bb34c P 73042c6c0e8a4affa6e460bb19d7c5c9: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "73042c6c0e8a4affa6e460bb19d7c5c9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5124d225ee724a998f64980e62a4dd51" member_type: VOTER last_known_addr { host: "127.11.102.1" port: 33471 } } peers { permanent_uuid: "73042c6c0e8a4affa6e460bb19d7c5c9" member_type: VOTER last_known_addr { host: "127.11.102.2" port: 43331 } } peers { permanent_uuid: "e61acac2279e47ffbb9af68a1df4d31d" member_type: VOTER last_known_addr { host: "127.11.102.3" port: 38435 } } }
I20260620 06:06:52.588372 13938 tablet_replica.cc:442] T 04d800ddb0a04b3db00030c30e1bb34c P 73042c6c0e8a4affa6e460bb19d7c5c9: This TxnStatusTablet replica's current role is: LEADER
I20260620 06:06:52.597633 13905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:06:52.593634 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39924 (local address 127.11.102.3:38435)
0620 06:06:52.593754 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0620 06:06:52.593759 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 06:06:52.593774 (+ 15us) server_negotiation.cc:407] Connection header received
0620 06:06:52.593888 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:06:52.593891 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:06:52.593951 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:06:52.594096 (+ 145us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 06:06:52.594305 (+ 209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:06:52.595081 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:06:52.596084 (+ 1003us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:06:52.596307 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:06:52.596420 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:06:52.596701 (+ 281us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 06:06:52.597236 (+ 535us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 06:06:52.597356 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0620 06:06:52.597420 (+ 64us) server_negotiation.cc:299] Negotiation successful
0620 06:06:52.597475 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260620 06:06:52.608055 13903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:06:52.604833 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.102.3:50711 (local address 127.11.102.2:43331)
0620 06:06:52.604949 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0620 06:06:52.604953 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 06:06:52.605067 (+ 114us) server_negotiation.cc:407] Connection header received
0620 06:06:52.605219 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:06:52.605222 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:06:52.605272 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:06:52.605357 (+ 85us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 06:06:52.605592 (+ 235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:06:52.606107 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:06:52.606946 (+ 839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:06:52.607181 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:06:52.607291 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:06:52.607601 (+ 310us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 06:06:52.607712 (+ 111us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 06:06:52.607784 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0620 06:06:52.607869 (+ 85us) server_negotiation.cc:299] Negotiation successful
0620 06:06:52.607919 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260620 06:06:52.608059 13889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:06:52.604759 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.102.2:43331 (local address 127.11.102.3:50711)
0620 06:06:52.604987 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0620 06:06:52.605007 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 06:06:52.605112 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:06:52.605365 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:06:52.605368 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:06:52.605374 (+ 6us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 06:06:52.605490 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:06:52.605496 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:06:52.606226 (+ 730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:06:52.606229 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:06:52.606824 (+ 595us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:06:52.606831 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:06:52.607300 (+ 469us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:06:52.607304 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:06:52.607405 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:06:52.607445 (+ 40us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 06:06:52.607790 (+ 345us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 06:06:52.607798 (+ 8us) client_negotiation.cc:769] Sending connection context
0620 06:06:52.607862 (+ 64us) client_negotiation.cc:240] Negotiation successful
0620 06:06:52.607943 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129}
I20260620 06:06:52.616385 13557 raft_consensus.cc:1261] T ab6fcfff344d4968917c9d5991f20398 P 5124d225ee724a998f64980e62a4dd51 [term 1 FOLLOWER]: Refusing update from remote peer e61acac2279e47ffbb9af68a1df4d31d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 06:06:52.616590 13690 raft_consensus.cc:1261] T ab6fcfff344d4968917c9d5991f20398 P 73042c6c0e8a4affa6e460bb19d7c5c9 [term 1 FOLLOWER]: Refusing update from remote peer e61acac2279e47ffbb9af68a1df4d31d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 06:06:52.616825 13916 consensus_queue.cc:1035] T ab6fcfff344d4968917c9d5991f20398 P e61acac2279e47ffbb9af68a1df4d31d [LEADER]: Connected to new peer: Peer: permanent_uuid: "5124d225ee724a998f64980e62a4dd51" member_type: VOTER last_known_addr { host: "127.11.102.1" port: 33471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 06:06:52.617065 13912 consensus_queue.cc:1035] T ab6fcfff344d4968917c9d5991f20398 P e61acac2279e47ffbb9af68a1df4d31d [LEADER]: Connected to new peer: Peer: permanent_uuid: "73042c6c0e8a4affa6e460bb19d7c5c9" member_type: VOTER last_known_addr { host: "127.11.102.2" port: 43331 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 06:06:52.624758 13933 mvcc.cc:205] Tried to move back new op lower bound from 7298808269272928256 to 7298808268499648512. Current Snapshot: MvccSnapshot[applied={T|T < 7298808269272928256}]
I20260620 06:06:52.640818 13949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:06:52.636872 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.102.3:38435 (local address 127.11.102.2:43927)
0620 06:06:52.637237 (+ 365us) negotiation.cc:107] Waiting for socket to connect
0620 06:06:52.637253 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 06:06:52.637356 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:06:52.637580 (+ 224us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:06:52.637583 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:06:52.637603 (+ 20us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 06:06:52.637744 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:06:52.637750 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:06:52.638781 (+ 1031us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:06:52.638787 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:06:52.639540 (+ 753us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:06:52.639550 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:06:52.640058 (+ 508us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:06:52.640061 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:06:52.640160 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:06:52.640209 (+ 49us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 06:06:52.640521 (+ 312us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 06:06:52.640529 (+ 8us) client_negotiation.cc:769] Sending connection context
0620 06:06:52.640589 (+ 60us) client_negotiation.cc:240] Negotiation successful
0620 06:06:52.640677 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":239,"thread_start_us":149,"threads_started":1}
I20260620 06:06:52.640805 13905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:06:52.636916 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.102.2:43927 (local address 127.11.102.3:38435)
0620 06:06:52.637059 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0620 06:06:52.637064 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 06:06:52.637326 (+ 262us) server_negotiation.cc:407] Connection header received
0620 06:06:52.637442 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:06:52.637445 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:06:52.637488 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:06:52.637550 (+ 62us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 06:06:52.637975 (+ 425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:06:52.638678 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:06:52.639700 (+ 1022us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:06:52.639946 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:06:52.640059 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:06:52.640340 (+ 281us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 06:06:52.640428 (+ 88us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 06:06:52.640507 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0620 06:06:52.640599 (+ 92us) server_negotiation.cc:299] Negotiation successful
0620 06:06:52.640661 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260620 06:06:52.670325 13399 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57814:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 06:06:52.670511 13399 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57814:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 06:06:52.673112 13399 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 794f593770bb429487ad0ec805dae6bf: Sending DeleteTablet for 3 replicas of tablet ab6fcfff344d4968917c9d5991f20398
I20260620 06:06:52.674010 13537 tablet_service.cc:1514] Processing DeleteTablet for tablet ab6fcfff344d4968917c9d5991f20398 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 06:06:52 UTC) from {username='kudu', principal='kudu/127.11.102.62@KRBTEST.COM'} at 127.0.0.1:56016
I20260620 06:06:52.674515 13952 tablet_replica.cc:330] T ab6fcfff344d4968917c9d5991f20398 P 5124d225ee724a998f64980e62a4dd51: stopping tablet replica
I20260620 06:06:52.674451 13399 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57814:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260620 06:06:52.674515 13677 tablet_service.cc:1514] Processing DeleteTablet for tablet ab6fcfff344d4968917c9d5991f20398 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 06:06:52 UTC) from {username='kudu', principal='kudu/127.11.102.62@KRBTEST.COM'} at 127.0.0.1:50158
W20260620 06:06:52.674817 13399 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 06:06:52.675041 13817 tablet_service.cc:1514] Processing DeleteTablet for tablet ab6fcfff344d4968917c9d5991f20398 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 06:06:52 UTC) from {username='kudu', principal='kudu/127.11.102.62@KRBTEST.COM'} at 127.0.0.1:39920
I20260620 06:06:52.675129 13952 raft_consensus.cc:2229] T ab6fcfff344d4968917c9d5991f20398 P 5124d225ee724a998f64980e62a4dd51 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 06:06:52.676368 13952 raft_consensus.cc:2258] T ab6fcfff344d4968917c9d5991f20398 P 5124d225ee724a998f64980e62a4dd51 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 06:06:52.678040 13952 ts_tablet_manager.cc:1905] T ab6fcfff344d4968917c9d5991f20398 P 5124d225ee724a998f64980e62a4dd51: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 06:06:52.679831 13537 tablet_service.cc:1467] Processing CreateTablet for tablet bb3c431767394dda86fa299333a5b1eb (DEFAULT_TABLE table=test-table [id=e748cf945e024425b7477cc9ce605ea1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 06:06:52.680114 13537 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bb3c431767394dda86fa299333a5b1eb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 06:06:52.680061 13817 tablet_service.cc:1467] Processing CreateTablet for tablet bb3c431767394dda86fa299333a5b1eb (DEFAULT_TABLE table=test-table [id=e748cf945e024425b7477cc9ce605ea1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 06:06:52.680296 13817 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bb3c431767394dda86fa299333a5b1eb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 06:06:52.681496 13952 ts_tablet_manager.cc:1918] T ab6fcfff344d4968917c9d5991f20398 P 5124d225ee724a998f64980e62a4dd51: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 06:06:52.681632 13952 log.cc:1196] T ab6fcfff344d4968917c9d5991f20398 P 5124d225ee724a998f64980e62a4dd51: Deleting WAL directory at /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781935603120653-11672-0/minicluster-data/ts-0/wal/wals/ab6fcfff344d4968917c9d5991f20398
I20260620 06:06:52.682019 13952 ts_tablet_manager.cc:1939] T ab6fcfff344d4968917c9d5991f20398 P 5124d225ee724a998f64980e62a4dd51: Deleting consensus metadata
I20260620 06:06:52.682454 13908 tablet_bootstrap.cc:492] T bb3c431767394dda86fa299333a5b1eb P 5124d225ee724a998f64980e62a4dd51: Bootstrap starting.
I20260620 06:06:52.682780 13906 tablet_bootstrap.cc:492] T bb3c431767394dda86fa299333a5b1eb P e61acac2279e47ffbb9af68a1df4d31d: Bootstrap starting.
I20260620 06:06:52.683101 13953 tablet_replica.cc:330] T ab6fcfff344d4968917c9d5991f20398 P 73042c6c0e8a4affa6e460bb19d7c5c9: stopping tablet replica
I20260620 06:06:52.683377 13908 tablet_bootstrap.cc:654] T bb3c431767394dda86fa299333a5b1eb P 5124d225ee724a998f64980e62a4dd51: Neither blocks nor log segments found. Creating new log.
I20260620 06:06:52.683437 13953 raft_consensus.cc:2229] T ab6fcfff344d4968917c9d5991f20398 P 73042c6c0e8a4affa6e460bb19d7c5c9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 06:06:52.683641 13953 raft_consensus.cc:2258] T ab6fcfff344d4968917c9d5991f20398 P 73042c6c0e8a4affa6e460bb19d7c5c9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 06:06:52.683756 13906 tablet_bootstrap.cc:654] T bb3c431767394dda86fa299333a5b1eb P e61acac2279e47ffbb9af68a1df4d31d: Neither blocks nor log segments found. Creating new log.
I20260620 06:06:52.683947 13954 tablet_replica.cc:330] T ab6fcfff344d4968917c9d5991f20398 P e61acac2279e47ffbb9af68a1df4d31d: stopping tablet replica
I20260620 06:06:52.684062 13387 catalog_manager.cc:4841] TS 5124d225ee724a998f64980e62a4dd51 (127.11.102.1:33471): tablet ab6fcfff344d4968917c9d5991f20398 (table test-table [id=25eb0e3072b9489bb49f008916c8bc70]) successfully deleted
I20260620 06:06:52.684234 13954 raft_consensus.cc:2229] T ab6fcfff344d4968917c9d5991f20398 P e61acac2279e47ffbb9af68a1df4d31d [term 1 LEADER]: Raft consensus shutting down.
I20260620 06:06:52.684599 13954 raft_consensus.cc:2258] T ab6fcfff344d4968917c9d5991f20398 P e61acac2279e47ffbb9af68a1df4d31d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 06:06:52.685128 13677 tablet_service.cc:1467] Processing CreateTablet for tablet bb3c431767394dda86fa299333a5b1eb (DEFAULT_TABLE table=test-table [id=e748cf945e024425b7477cc9ce605ea1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 06:06:52.685382 13677 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bb3c431767394dda86fa299333a5b1eb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 06:06:52.686069 13953 ts_tablet_manager.cc:1905] T ab6fcfff344d4968917c9d5991f20398 P 73042c6c0e8a4affa6e460bb19d7c5c9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 06:06:52.688063 13907 tablet_bootstrap.cc:492] T bb3c431767394dda86fa299333a5b1eb P 73042c6c0e8a4affa6e460bb19d7c5c9: Bootstrap starting.
I20260620 06:06:52.689004 13907 tablet_bootstrap.cc:654] T bb3c431767394dda86fa299333a5b1eb P 73042c6c0e8a4affa6e460bb19d7c5c9: Neither blocks nor log segments found. Creating new log.
I20260620 06:06:52.689046 13953 ts_tablet_manager.cc:1918] T ab6fcfff344d4968917c9d5991f20398 P 73042c6c0e8a4affa6e460bb19d7c5c9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 06:06:52.689116 13953 log.cc:1196] T ab6fcfff344d4968917c9d5991f20398 P 73042c6c0e8a4affa6e460bb19d7c5c9: Deleting WAL directory at /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781935603120653-11672-0/minicluster-data/ts-1/wal/wals/ab6fcfff344d4968917c9d5991f20398
I20260620 06:06:52.689445 13953 ts_tablet_manager.cc:1939] T ab6fcfff344d4968917c9d5991f20398 P 73042c6c0e8a4affa6e460bb19d7c5c9: Deleting consensus metadata
I20260620 06:06:52.690300 13907 tablet_bootstrap.cc:492] T bb3c431767394dda86fa299333a5b1eb P 73042c6c0e8a4affa6e460bb19d7c5c9: No bootstrap required, opened a new log
I20260620 06:06:52.690402 13907 ts_tablet_manager.cc:1397] T bb3c431767394dda86fa299333a5b1eb P 73042c6c0e8a4affa6e460bb19d7c5c9: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260620 06:06:52.690963 13907 raft_consensus.cc:348] T bb3c431767394dda86fa299333a5b1eb P 73042c6c0e8a4affa6e460bb19d7c5c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5124d225ee724a998f64980e62a4dd51" member_type: VOTER last_known_addr { host: "127.11.102.1" port: 33471 } } peers { permanent_uuid: "73042c6c0e8a4affa6e460bb19d7c5c9" member_type: VOTER last_known_addr { host: "127.11.102.2" port: 43331 } } peers { permanent_uuid: "e61acac2279e47ffbb9af68a1df4d31d" member_type: VOTER last_known_addr { host: "127.11.102.3" port: 38435 } }
I20260620 06:06:52.691077 13907 raft_consensus.cc:374] T bb3c431767394dda86fa299333a5b1eb P 73042c6c0e8a4affa6e460bb19d7c5c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 06:06:52.691105 13907 raft_consensus.cc:729] T bb3c431767394dda86fa299333a5b1eb P 73042c6c0e8a4affa6e460bb19d7c5c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73042c6c0e8a4affa6e460bb19d7c5c9, State: Initialized, Role: FOLLOWER
I20260620 06:06:52.691334 13906 tablet_bootstrap.cc:492] T bb3c431767394dda86fa299333a5b1eb P e61acac2279e47ffbb9af68a1df4d31d: No bootstrap required, opened a new log
I20260620 06:06:52.691349 13954 ts_tablet_manager.cc:1905] T ab6fcfff344d4968917c9d5991f20398 P e61acac2279e47ffbb9af68a1df4d31d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 06:06:52.691447 13906 ts_tablet_manager.cc:1397] T bb3c431767394dda86fa299333a5b1eb P e61acac2279e47ffbb9af68a1df4d31d: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.000s
I20260620 06:06:52.691781 13907 consensus_queue.cc:260] T bb3c431767394dda86fa299333a5b1eb P 73042c6c0e8a4affa6e460bb19d7c5c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5124d225ee724a998f64980e62a4dd51" member_type: VOTER last_known_addr { host: "127.11.102.1" port: 33471 } } peers { permanent_uuid: "73042c6c0e8a4affa6e460bb19d7c5c9" member_type: VOTER last_known_addr { host: "127.11.102.2" port: 43331 } } peers { permanent_uuid: "e61acac2279e47ffbb9af68a1df4d31d" member_type: VOTER last_known_addr { host: "127.11.102.3" port: 38435 } }
I20260620 06:06:52.691918 13906 raft_consensus.cc:348] T bb3c431767394dda86fa299333a5b1eb P e61acac2279e47ffbb9af68a1df4d31d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5124d225ee724a998f64980e62a4dd51" member_type: VOTER last_known_addr { host: "127.11.102.1" port: 33471 } } peers { permanent_uuid: "73042c6c0e8a4affa6e460bb19d7c5c9" member_type: VOTER last_known_addr { host: "127.11.102.2" port: 43331 } } peers { permanent_uuid: "e61acac2279e47ffbb9af68a1df4d31d" member_type: VOTER last_known_addr { host: "127.11.102.3" port: 38435 } }
I20260620 06:06:52.692027 13906 raft_consensus.cc:374] T bb3c431767394dda86fa299333a5b1eb P e61acac2279e47ffbb9af68a1df4d31d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 06:06:52.692058 13906 raft_consensus.cc:729] T bb3c431767394dda86fa299333a5b1eb P e61acac2279e47ffbb9af68a1df4d31d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e61acac2279e47ffbb9af68a1df4d31d, State: Initialized, Role: FOLLOWER
I20260620 06:06:52.692157 13907 ts_tablet_manager.cc:1428] T bb3c431767394dda86fa299333a5b1eb P 73042c6c0e8a4affa6e460bb19d7c5c9: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260620 06:06:52.692166 13906 consensus_queue.cc:260] T bb3c431767394dda86fa299333a5b1eb P e61acac2279e47ffbb9af68a1df4d31d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5124d225ee724a998f64980e62a4dd51" member_type: VOTER last_known_addr { host: "127.11.102.1" port: 33471 } } peers { permanent_uuid: "73042c6c0e8a4affa6e460bb19d7c5c9" member_type: VOTER last_known_addr { host: "127.11.102.2" port: 43331 } } peers { permanent_uuid: "e61acac2279e47ffbb9af68a1df4d31d" member_type: VOTER last_known_addr { host: "127.11.102.3" port: 38435 } }
I20260620 06:06:52.692394 13906 ts_tablet_manager.cc:1428] T bb3c431767394dda86fa299333a5b1eb P e61acac2279e47ffbb9af68a1df4d31d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260620 06:06:52.694182 13954 ts_tablet_manager.cc:1918] T ab6fcfff344d4968917c9d5991f20398 P e61acac2279e47ffbb9af68a1df4d31d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 06:06:52.694279 13954 log.cc:1196] T ab6fcfff344d4968917c9d5991f20398 P e61acac2279e47ffbb9af68a1df4d31d: Deleting WAL directory at /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781935603120653-11672-0/minicluster-data/ts-2/wal/wals/ab6fcfff344d4968917c9d5991f20398
I20260620 06:06:52.694592 13954 ts_tablet_manager.cc:1939] T ab6fcfff344d4968917c9d5991f20398 P e61acac2279e47ffbb9af68a1df4d31d: Deleting consensus metadata
I20260620 06:06:52.695631 13386 catalog_manager.cc:4841] TS e61acac2279e47ffbb9af68a1df4d31d (127.11.102.3:38435): tablet ab6fcfff344d4968917c9d5991f20398 (table test-table [id=25eb0e3072b9489bb49f008916c8bc70]) successfully deleted
I20260620 06:06:52.704319 13384 catalog_manager.cc:4841] TS 73042c6c0e8a4affa6e460bb19d7c5c9 (127.11.102.2:43331): tablet ab6fcfff344d4968917c9d5991f20398 (table test-table [id=25eb0e3072b9489bb49f008916c8bc70]) successfully deleted
I20260620 06:06:52.705036 13908 tablet_bootstrap.cc:492] T bb3c431767394dda86fa299333a5b1eb P 5124d225ee724a998f64980e62a4dd51: No bootstrap required, opened a new log
I20260620 06:06:52.705140 13908 ts_tablet_manager.cc:1397] T bb3c431767394dda86fa299333a5b1eb P 5124d225ee724a998f64980e62a4dd51: Time spent bootstrapping tablet: real 0.023s user 0.000s sys 0.002s
I20260620 06:06:52.705603 13908 raft_consensus.cc:348] T bb3c431767394dda86fa299333a5b1eb P 5124d225ee724a998f64980e62a4dd51 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5124d225ee724a998f64980e62a4dd51" member_type: VOTER last_known_addr { host: "127.11.102.1" port: 33471 } } peers { permanent_uuid: "73042c6c0e8a4affa6e460bb19d7c5c9" member_type: VOTER last_known_addr { host: "127.11.102.2" port: 43331 } } peers { permanent_uuid: "e61acac2279e47ffbb9af68a1df4d31d" member_type: VOTER last_known_addr { host: "127.11.102.3" port: 38435 } }
I20260620 06:06:52.705747 13908 raft_consensus.cc:374] T bb3c431767394dda86fa299333a5b1eb P 5124d225ee724a998f64980e62a4dd51 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 06:06:52.705789 13908 raft_consensus.cc:729] T bb3c431767394dda86fa299333a5b1eb P 5124d225ee724a998f64980e62a4dd51 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5124d225ee724a998f64980e62a4dd51, State: Initialized, Role: FOLLOWER
I20260620 06:06:52.705926 13908 consensus_queue.cc:260] T bb3c431767394dda86fa299333a5b1eb P 5124d225ee724a998f64980e62a4dd51 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5124d225ee724a998f64980e62a4dd51" member_type: VOTER last_known_addr { host: "127.11.102.1" port: 33471 } } peers { permanent_uuid: "73042c6c0e8a4affa6e460bb19d7c5c9" member_type: VOTER last_known_addr { host: "127.11.102.2" port: 43331 } } peers { permanent_uuid: "e61acac2279e47ffbb9af68a1df4d31d" member_type: VOTER last_known_addr { host: "127.11.102.3" port: 38435 } }
I20260620 06:06:52.706187 13908 ts_tablet_manager.cc:1428] T bb3c431767394dda86fa299333a5b1eb P 5124d225ee724a998f64980e62a4dd51: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260620 06:06:52.833876 13928 raft_consensus.cc:482] T bb3c431767394dda86fa299333a5b1eb P 73042c6c0e8a4affa6e460bb19d7c5c9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 06:06:52.834033 13928 raft_consensus.cc:504] T bb3c431767394dda86fa299333a5b1eb P 73042c6c0e8a4affa6e460bb19d7c5c9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5124d225ee724a998f64980e62a4dd51" member_type: VOTER last_known_addr { host: "127.11.102.1" port: 33471 } } peers { permanent_uuid: "73042c6c0e8a4affa6e460bb19d7c5c9" member_type: VOTER last_known_addr { host: "127.11.102.2" port: 43331 } } peers { permanent_uuid: "e61acac2279e47ffbb9af68a1df4d31d" member_type: VOTER last_known_addr { host: "127.11.102.3" port: 38435 } }
I20260620 06:06:52.834507 13928 leader_election.cc:288] T bb3c431767394dda86fa299333a5b1eb P 73042c6c0e8a4affa6e460bb19d7c5c9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5124d225ee724a998f64980e62a4dd51 (127.11.102.1:33471), e61acac2279e47ffbb9af68a1df4d31d (127.11.102.3:38435)
I20260620 06:06:52.834977 13837 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bb3c431767394dda86fa299333a5b1eb" candidate_uuid: "73042c6c0e8a4affa6e460bb19d7c5c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e61acac2279e47ffbb9af68a1df4d31d" is_pre_election: true
I20260620 06:06:52.835145 13837 raft_consensus.cc:2454] T bb3c431767394dda86fa299333a5b1eb P e61acac2279e47ffbb9af68a1df4d31d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 73042c6c0e8a4affa6e460bb19d7c5c9 in term 0.
I20260620 06:06:52.835431 13557 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bb3c431767394dda86fa299333a5b1eb" candidate_uuid: "73042c6c0e8a4affa6e460bb19d7c5c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5124d225ee724a998f64980e62a4dd51" is_pre_election: true
I20260620 06:06:52.835492 13630 leader_election.cc:302] T bb3c431767394dda86fa299333a5b1eb P 73042c6c0e8a4affa6e460bb19d7c5c9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 73042c6c0e8a4affa6e460bb19d7c5c9, e61acac2279e47ffbb9af68a1df4d31d; no voters:
I20260620 06:06:52.835577 13557 raft_consensus.cc:2454] T bb3c431767394dda86fa299333a5b1eb P 5124d225ee724a998f64980e62a4dd51 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 73042c6c0e8a4affa6e460bb19d7c5c9 in term 0.
I20260620 06:06:52.835698 13928 raft_consensus.cc:2781] T bb3c431767394dda86fa299333a5b1eb P 73042c6c0e8a4affa6e460bb19d7c5c9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 06:06:52.835769 13928 raft_consensus.cc:482] T bb3c431767394dda86fa299333a5b1eb P 73042c6c0e8a4affa6e460bb19d7c5c9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 06:06:52.835810 13928 raft_consensus.cc:3037] T bb3c431767394dda86fa299333a5b1eb P 73042c6c0e8a4affa6e460bb19d7c5c9 [term 0 FOLLOWER]: Advancing to term 1
I20260620 06:06:52.837041 13928 raft_consensus.cc:504] T bb3c431767394dda86fa299333a5b1eb P 73042c6c0e8a4affa6e460bb19d7c5c9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5124d225ee724a998f64980e62a4dd51" member_type: VOTER last_known_addr { host: "127.11.102.1" port: 33471 } } peers { permanent_uuid: "73042c6c0e8a4affa6e460bb19d7c5c9" member_type: VOTER last_known_addr { host: "127.11.102.2" port: 43331 } } peers { permanent_uuid: "e61acac2279e47ffbb9af68a1df4d31d" member_type: VOTER last_known_addr { host: "127.11.102.3" port: 38435 } }
I20260620 06:06:52.837460 13928 leader_election.cc:288] T bb3c431767394dda86fa299333a5b1eb P 73042c6c0e8a4affa6e460bb19d7c5c9 [CANDIDATE]: Term 1 election: Requested vote from peers 5124d225ee724a998f64980e62a4dd51 (127.11.102.1:33471), e61acac2279e47ffbb9af68a1df4d31d (127.11.102.3:38435)
I20260620 06:06:52.837767 13837 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bb3c431767394dda86fa299333a5b1eb" candidate_uuid: "73042c6c0e8a4affa6e460bb19d7c5c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e61acac2279e47ffbb9af68a1df4d31d"
I20260620 06:06:52.837868 13557 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bb3c431767394dda86fa299333a5b1eb" candidate_uuid: "73042c6c0e8a4affa6e460bb19d7c5c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5124d225ee724a998f64980e62a4dd51"
I20260620 06:06:52.837942 13837 raft_consensus.cc:3037] T bb3c431767394dda86fa299333a5b1eb P e61acac2279e47ffbb9af68a1df4d31d [term 0 FOLLOWER]: Advancing to term 1
I20260620 06:06:52.837987 13557 raft_consensus.cc:3037] T bb3c431767394dda86fa299333a5b1eb P 5124d225ee724a998f64980e62a4dd51 [term 0 FOLLOWER]: Advancing to term 1
I20260620 06:06:52.838828 13837 raft_consensus.cc:2454] T bb3c431767394dda86fa299333a5b1eb P e61acac2279e47ffbb9af68a1df4d31d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 73042c6c0e8a4affa6e460bb19d7c5c9 in term 1.
I20260620 06:06:52.838899 13557 raft_consensus.cc:2454] T bb3c431767394dda86fa299333a5b1eb P 5124d225ee724a998f64980e62a4dd51 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 73042c6c0e8a4affa6e460bb19d7c5c9 in term 1.
I20260620 06:06:52.839131 13630 leader_election.cc:302] T bb3c431767394dda86fa299333a5b1eb P 73042c6c0e8a4affa6e460bb19d7c5c9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 73042c6c0e8a4affa6e460bb19d7c5c9, e61acac2279e47ffbb9af68a1df4d31d; no voters:
I20260620 06:06:52.839403 13928 raft_consensus.cc:2781] T bb3c431767394dda86fa299333a5b1eb P 73042c6c0e8a4affa6e460bb19d7c5c9 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 06:06:52.839514 13928 raft_consensus.cc:686] T bb3c431767394dda86fa299333a5b1eb P 73042c6c0e8a4affa6e460bb19d7c5c9 [term 1 LEADER]: Becoming Leader. State: Replica: 73042c6c0e8a4affa6e460bb19d7c5c9, State: Running, Role: LEADER
I20260620 06:06:52.839675 13928 consensus_queue.cc:237] T bb3c431767394dda86fa299333a5b1eb P 73042c6c0e8a4affa6e460bb19d7c5c9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5124d225ee724a998f64980e62a4dd51" member_type: VOTER last_known_addr { host: "127.11.102.1" port: 33471 } } peers { permanent_uuid: "73042c6c0e8a4affa6e460bb19d7c5c9" member_type: VOTER last_known_addr { host: "127.11.102.2" port: 43331 } } peers { permanent_uuid: "e61acac2279e47ffbb9af68a1df4d31d" member_type: VOTER last_known_addr { host: "127.11.102.3" port: 38435 } }
I20260620 06:06:52.841351 13396 catalog_manager.cc:5495] T bb3c431767394dda86fa299333a5b1eb P 73042c6c0e8a4affa6e460bb19d7c5c9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 73042c6c0e8a4affa6e460bb19d7c5c9 (127.11.102.2). New cstate: current_term: 1 leader_uuid: "73042c6c0e8a4affa6e460bb19d7c5c9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5124d225ee724a998f64980e62a4dd51" member_type: VOTER last_known_addr { host: "127.11.102.1" port: 33471 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "73042c6c0e8a4affa6e460bb19d7c5c9" member_type: VOTER last_known_addr { host: "127.11.102.2" port: 43331 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e61acac2279e47ffbb9af68a1df4d31d" member_type: VOTER last_known_addr { host: "127.11.102.3" port: 38435 } health_report { overall_health: UNKNOWN } } }
I20260620 06:06:52.889245 13903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:06:52.885974 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50178 (local address 127.11.102.2:43331)
0620 06:06:52.886090 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0620 06:06:52.886094 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 06:06:52.886126 (+ 32us) server_negotiation.cc:407] Connection header received
0620 06:06:52.886240 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:06:52.886242 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:06:52.886280 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:06:52.886342 (+ 62us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 06:06:52.886597 (+ 255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:06:52.887119 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:06:52.888295 (+ 1176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:06:52.888509 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:06:52.888603 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:06:52.888884 (+ 281us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 06:06:52.888948 (+ 64us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 06:06:52.889002 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0620 06:06:52.889108 (+ 106us) server_negotiation.cc:299] Negotiation successful
0620 06:06:52.889147 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260620 06:06:52.897542 13949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:06:52.894639 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.102.2:43331 (local address 127.11.102.2:45075)
0620 06:06:52.894772 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0620 06:06:52.894787 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 06:06:52.894893 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:06:52.895120 (+ 227us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:06:52.895122 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:06:52.895128 (+ 6us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 06:06:52.895208 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:06:52.895213 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:06:52.895950 (+ 737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:06:52.895953 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:06:52.896526 (+ 573us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:06:52.896533 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:06:52.897015 (+ 482us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:06:52.897017 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:06:52.897077 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:06:52.897100 (+ 23us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 06:06:52.897322 (+ 222us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 06:06:52.897327 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 06:06:52.897379 (+ 52us) client_negotiation.cc:240] Negotiation successful
0620 06:06:52.897444 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":47}
I20260620 06:06:52.897732 13903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:06:52.894699 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.102.2:45075 (local address 127.11.102.2:43331)
0620 06:06:52.894822 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0620 06:06:52.894826 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 06:06:52.894838 (+ 12us) server_negotiation.cc:407] Connection header received
0620 06:06:52.894990 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:06:52.894993 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:06:52.895029 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:06:52.895104 (+ 75us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 06:06:52.895300 (+ 196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:06:52.895845 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:06:52.896656 (+ 811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:06:52.896910 (+ 254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:06:52.896991 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:06:52.897194 (+ 203us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 06:06:52.897250 (+ 56us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 06:06:52.897300 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0620 06:06:52.897558 (+ 258us) server_negotiation.cc:299] Negotiation successful
0620 06:06:52.897605 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260620 06:06:52.902952 13837 raft_consensus.cc:1261] T bb3c431767394dda86fa299333a5b1eb P e61acac2279e47ffbb9af68a1df4d31d [term 1 FOLLOWER]: Refusing update from remote peer 73042c6c0e8a4affa6e460bb19d7c5c9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 06:06:52.903005 13557 raft_consensus.cc:1261] T bb3c431767394dda86fa299333a5b1eb P 5124d225ee724a998f64980e62a4dd51 [term 1 FOLLOWER]: Refusing update from remote peer 73042c6c0e8a4affa6e460bb19d7c5c9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 06:06:52.903384 13928 consensus_queue.cc:1035] T bb3c431767394dda86fa299333a5b1eb P 73042c6c0e8a4affa6e460bb19d7c5c9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5124d225ee724a998f64980e62a4dd51" member_type: VOTER last_known_addr { host: "127.11.102.1" port: 33471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 06:06:52.903534 13938 consensus_queue.cc:1035] T bb3c431767394dda86fa299333a5b1eb P 73042c6c0e8a4affa6e460bb19d7c5c9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e61acac2279e47ffbb9af68a1df4d31d" member_type: VOTER last_known_addr { host: "127.11.102.3" port: 38435 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260620 06:06:52.916005 13933 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260620 06:06:54.467730 13399 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57814:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 06:06:54.467909 13399 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57814:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 06:06:54.470243 13399 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 794f593770bb429487ad0ec805dae6bf: Sending DeleteTablet for 3 replicas of tablet bb3c431767394dda86fa299333a5b1eb
I20260620 06:06:54.470722 13676 tablet_service.cc:1514] Processing DeleteTablet for tablet bb3c431767394dda86fa299333a5b1eb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 06:06:54 UTC) from {username='kudu', principal='kudu/127.11.102.62@KRBTEST.COM'} at 127.0.0.1:50158
I20260620 06:06:54.470734 13537 tablet_service.cc:1514] Processing DeleteTablet for tablet bb3c431767394dda86fa299333a5b1eb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 06:06:54 UTC) from {username='kudu', principal='kudu/127.11.102.62@KRBTEST.COM'} at 127.0.0.1:56016
I20260620 06:06:54.470808 13817 tablet_service.cc:1514] Processing DeleteTablet for tablet bb3c431767394dda86fa299333a5b1eb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 06:06:54 UTC) from {username='kudu', principal='kudu/127.11.102.62@KRBTEST.COM'} at 127.0.0.1:39920
I20260620 06:06:54.471036 13984 tablet_replica.cc:330] T bb3c431767394dda86fa299333a5b1eb P 5124d225ee724a998f64980e62a4dd51: stopping tablet replica
I20260620 06:06:54.471118 13986 tablet_replica.cc:330] T bb3c431767394dda86fa299333a5b1eb P e61acac2279e47ffbb9af68a1df4d31d: stopping tablet replica
I20260620 06:06:54.471182 13984 raft_consensus.cc:2229] T bb3c431767394dda86fa299333a5b1eb P 5124d225ee724a998f64980e62a4dd51 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 06:06:54.471247 13986 raft_consensus.cc:2229] T bb3c431767394dda86fa299333a5b1eb P e61acac2279e47ffbb9af68a1df4d31d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 06:06:54.471294 13984 raft_consensus.cc:2258] T bb3c431767394dda86fa299333a5b1eb P 5124d225ee724a998f64980e62a4dd51 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 06:06:54.471359 13986 raft_consensus.cc:2258] T bb3c431767394dda86fa299333a5b1eb P e61acac2279e47ffbb9af68a1df4d31d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 06:06:54.471493 13985 tablet_replica.cc:330] T bb3c431767394dda86fa299333a5b1eb P 73042c6c0e8a4affa6e460bb19d7c5c9: stopping tablet replica
I20260620 06:06:54.471628 13985 raft_consensus.cc:2229] T bb3c431767394dda86fa299333a5b1eb P 73042c6c0e8a4affa6e460bb19d7c5c9 [term 1 LEADER]: Raft consensus shutting down.
I20260620 06:06:54.471920 13985 raft_consensus.cc:2258] T bb3c431767394dda86fa299333a5b1eb P 73042c6c0e8a4affa6e460bb19d7c5c9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 06:06:54.472473 13984 ts_tablet_manager.cc:1905] T bb3c431767394dda86fa299333a5b1eb P 5124d225ee724a998f64980e62a4dd51: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 06:06:54.474236 13986 ts_tablet_manager.cc:1905] T bb3c431767394dda86fa299333a5b1eb P e61acac2279e47ffbb9af68a1df4d31d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 06:06:54.475035 13984 ts_tablet_manager.cc:1918] T bb3c431767394dda86fa299333a5b1eb P 5124d225ee724a998f64980e62a4dd51: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260620 06:06:54.475142 13984 log.cc:1196] T bb3c431767394dda86fa299333a5b1eb P 5124d225ee724a998f64980e62a4dd51: Deleting WAL directory at /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781935603120653-11672-0/minicluster-data/ts-0/wal/wals/bb3c431767394dda86fa299333a5b1eb
I20260620 06:06:54.475442 13984 ts_tablet_manager.cc:1939] T bb3c431767394dda86fa299333a5b1eb P 5124d225ee724a998f64980e62a4dd51: Deleting consensus metadata
I20260620 06:06:54.476128 13387 catalog_manager.cc:4841] TS 5124d225ee724a998f64980e62a4dd51 (127.11.102.1:33471): tablet bb3c431767394dda86fa299333a5b1eb (table test-table [id=e748cf945e024425b7477cc9ce605ea1]) successfully deleted
I20260620 06:06:54.476374 13985 ts_tablet_manager.cc:1905] T bb3c431767394dda86fa299333a5b1eb P 73042c6c0e8a4affa6e460bb19d7c5c9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 06:06:54.477093 13986 ts_tablet_manager.cc:1918] T bb3c431767394dda86fa299333a5b1eb P e61acac2279e47ffbb9af68a1df4d31d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260620 06:06:54.477185 13986 log.cc:1196] T bb3c431767394dda86fa299333a5b1eb P e61acac2279e47ffbb9af68a1df4d31d: Deleting WAL directory at /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781935603120653-11672-0/minicluster-data/ts-2/wal/wals/bb3c431767394dda86fa299333a5b1eb
I20260620 06:06:54.477411 13986 ts_tablet_manager.cc:1939] T bb3c431767394dda86fa299333a5b1eb P e61acac2279e47ffbb9af68a1df4d31d: Deleting consensus metadata
I20260620 06:06:54.478034 13386 catalog_manager.cc:4841] TS e61acac2279e47ffbb9af68a1df4d31d (127.11.102.3:38435): tablet bb3c431767394dda86fa299333a5b1eb (table test-table [id=e748cf945e024425b7477cc9ce605ea1]) successfully deleted
I20260620 06:06:54.478991 13985 ts_tablet_manager.cc:1918] T bb3c431767394dda86fa299333a5b1eb P 73042c6c0e8a4affa6e460bb19d7c5c9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260620 06:06:54.479082 13985 log.cc:1196] T bb3c431767394dda86fa299333a5b1eb P 73042c6c0e8a4affa6e460bb19d7c5c9: Deleting WAL directory at /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781935603120653-11672-0/minicluster-data/ts-1/wal/wals/bb3c431767394dda86fa299333a5b1eb
I20260620 06:06:54.479346 13985 ts_tablet_manager.cc:1939] T bb3c431767394dda86fa299333a5b1eb P 73042c6c0e8a4affa6e460bb19d7c5c9: Deleting consensus metadata
I20260620 06:06:54.480031 13384 catalog_manager.cc:4841] TS 73042c6c0e8a4affa6e460bb19d7c5c9 (127.11.102.2:43331): tablet bb3c431767394dda86fa299333a5b1eb (table test-table [id=e748cf945e024425b7477cc9ce605ea1]) successfully deleted
Jun 20 06:06:54 dist-test-slave-9hm5 krb5kdc[13355](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935614, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 20 06:06:54 dist-test-slave-9hm5 krb5kdc[13355](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935614, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.11.102.62@KRBTEST.COM
I20260620 06:06:54.497851 13994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:06:54.488927 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57842 (local address 127.11.102.62:39839)
0620 06:06:54.489217 (+ 290us) server_negotiation.cc:206] Beginning negotiation
0620 06:06:54.489229 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 06:06:54.489250 (+ 21us) server_negotiation.cc:407] Connection header received
0620 06:06:54.489430 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:06:54.489438 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:06:54.489481 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:06:54.489573 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:06:54.490282 (+ 709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:06:54.490827 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:06:54.491670 (+ 843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:06:54.491898 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:06:54.492036 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:06:54.494618 (+ 2582us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:06:54.494621 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:06:54.494623 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:06:54.494645 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:06:54.496126 (+ 1481us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:06:54.496596 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:06:54.496602 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:06:54.496609 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:06:54.496686 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:06:54.496967 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:06:54.496971 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:06:54.496972 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:06:54.497157 (+ 185us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:06:54.497297 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0620 06:06:54.497547 (+ 250us) server_negotiation.cc:299] Negotiation successful
0620 06:06:54.497675 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":214,"thread_start_us":90,"threads_started":1}
I20260620 06:06:54.500385 13399 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57842:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 06:06:54.500818 13399 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 06:06:54.505391 13537 tablet_service.cc:1467] Processing CreateTablet for tablet 0840f1fbda564ebba808038d7e2beb6f (DEFAULT_TABLE table=test-table [id=768e3479107f45278e1aa205078c2447]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 06:06:54.505553 13817 tablet_service.cc:1467] Processing CreateTablet for tablet 0840f1fbda564ebba808038d7e2beb6f (DEFAULT_TABLE table=test-table [id=768e3479107f45278e1aa205078c2447]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 06:06:54.505740 13537 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0840f1fbda564ebba808038d7e2beb6f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 06:06:54.505827 13817 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0840f1fbda564ebba808038d7e2beb6f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 06:06:54.507922 13995 tablet_bootstrap.cc:492] T 0840f1fbda564ebba808038d7e2beb6f P 5124d225ee724a998f64980e62a4dd51: Bootstrap starting.
I20260620 06:06:54.507977 13676 tablet_service.cc:1467] Processing CreateTablet for tablet 0840f1fbda564ebba808038d7e2beb6f (DEFAULT_TABLE table=test-table [id=768e3479107f45278e1aa205078c2447]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 06:06:54.508227 13996 tablet_bootstrap.cc:492] T 0840f1fbda564ebba808038d7e2beb6f P e61acac2279e47ffbb9af68a1df4d31d: Bootstrap starting.
I20260620 06:06:54.508265 13676 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0840f1fbda564ebba808038d7e2beb6f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 06:06:54.509042 13996 tablet_bootstrap.cc:654] T 0840f1fbda564ebba808038d7e2beb6f P e61acac2279e47ffbb9af68a1df4d31d: Neither blocks nor log segments found. Creating new log.
I20260620 06:06:54.509181 13995 tablet_bootstrap.cc:654] T 0840f1fbda564ebba808038d7e2beb6f P 5124d225ee724a998f64980e62a4dd51: Neither blocks nor log segments found. Creating new log.
I20260620 06:06:54.510197 13996 tablet_bootstrap.cc:492] T 0840f1fbda564ebba808038d7e2beb6f P e61acac2279e47ffbb9af68a1df4d31d: No bootstrap required, opened a new log
I20260620 06:06:54.510290 13996 ts_tablet_manager.cc:1397] T 0840f1fbda564ebba808038d7e2beb6f P e61acac2279e47ffbb9af68a1df4d31d: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260620 06:06:54.510331 13995 tablet_bootstrap.cc:492] T 0840f1fbda564ebba808038d7e2beb6f P 5124d225ee724a998f64980e62a4dd51: No bootstrap required, opened a new log
I20260620 06:06:54.510407 13995 ts_tablet_manager.cc:1397] T 0840f1fbda564ebba808038d7e2beb6f P 5124d225ee724a998f64980e62a4dd51: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260620 06:06:54.510736 13996 raft_consensus.cc:348] T 0840f1fbda564ebba808038d7e2beb6f P e61acac2279e47ffbb9af68a1df4d31d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5124d225ee724a998f64980e62a4dd51" member_type: VOTER last_known_addr { host: "127.11.102.1" port: 33471 } } peers { permanent_uuid: "73042c6c0e8a4affa6e460bb19d7c5c9" member_type: VOTER last_known_addr { host: "127.11.102.2" port: 43331 } } peers { permanent_uuid: "e61acac2279e47ffbb9af68a1df4d31d" member_type: VOTER last_known_addr { host: "127.11.102.3" port: 38435 } }
I20260620 06:06:54.510848 13996 raft_consensus.cc:374] T 0840f1fbda564ebba808038d7e2beb6f P e61acac2279e47ffbb9af68a1df4d31d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 06:06:54.510882 13996 raft_consensus.cc:729] T 0840f1fbda564ebba808038d7e2beb6f P e61acac2279e47ffbb9af68a1df4d31d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e61acac2279e47ffbb9af68a1df4d31d, State: Initialized, Role: FOLLOWER
I20260620 06:06:54.510987 13995 raft_consensus.cc:348] T 0840f1fbda564ebba808038d7e2beb6f P 5124d225ee724a998f64980e62a4dd51 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5124d225ee724a998f64980e62a4dd51" member_type: VOTER last_known_addr { host: "127.11.102.1" port: 33471 } } peers { permanent_uuid: "73042c6c0e8a4affa6e460bb19d7c5c9" member_type: VOTER last_known_addr { host: "127.11.102.2" port: 43331 } } peers { permanent_uuid: "e61acac2279e47ffbb9af68a1df4d31d" member_type: VOTER last_known_addr { host: "127.11.102.3" port: 38435 } }
I20260620 06:06:54.511090 13995 raft_consensus.cc:374] T 0840f1fbda564ebba808038d7e2beb6f P 5124d225ee724a998f64980e62a4dd51 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 06:06:54.511042 13996 consensus_queue.cc:260] T 0840f1fbda564ebba808038d7e2beb6f P e61acac2279e47ffbb9af68a1df4d31d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5124d225ee724a998f64980e62a4dd51" member_type: VOTER last_known_addr { host: "127.11.102.1" port: 33471 } } peers { permanent_uuid: "73042c6c0e8a4affa6e460bb19d7c5c9" member_type: VOTER last_known_addr { host: "127.11.102.2" port: 43331 } } peers { permanent_uuid: "e61acac2279e47ffbb9af68a1df4d31d" member_type: VOTER last_known_addr { host: "127.11.102.3" port: 38435 } }
I20260620 06:06:54.511121 13995 raft_consensus.cc:729] T 0840f1fbda564ebba808038d7e2beb6f P 5124d225ee724a998f64980e62a4dd51 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5124d225ee724a998f64980e62a4dd51, State: Initialized, Role: FOLLOWER
I20260620 06:06:54.511266 13995 consensus_queue.cc:260] T 0840f1fbda564ebba808038d7e2beb6f P 5124d225ee724a998f64980e62a4dd51 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5124d225ee724a998f64980e62a4dd51" member_type: VOTER last_known_addr { host: "127.11.102.1" port: 33471 } } peers { permanent_uuid: "73042c6c0e8a4affa6e460bb19d7c5c9" member_type: VOTER last_known_addr { host: "127.11.102.2" port: 43331 } } peers { permanent_uuid: "e61acac2279e47ffbb9af68a1df4d31d" member_type: VOTER last_known_addr { host: "127.11.102.3" port: 38435 } }
I20260620 06:06:54.511385 13999 tablet_bootstrap.cc:492] T 0840f1fbda564ebba808038d7e2beb6f P 73042c6c0e8a4affa6e460bb19d7c5c9: Bootstrap starting.
I20260620 06:06:54.511405 13996 ts_tablet_manager.cc:1428] T 0840f1fbda564ebba808038d7e2beb6f P e61acac2279e47ffbb9af68a1df4d31d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260620 06:06:54.511674 13995 ts_tablet_manager.cc:1428] T 0840f1fbda564ebba808038d7e2beb6f P 5124d225ee724a998f64980e62a4dd51: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260620 06:06:54.512217 13999 tablet_bootstrap.cc:654] T 0840f1fbda564ebba808038d7e2beb6f P 73042c6c0e8a4affa6e460bb19d7c5c9: Neither blocks nor log segments found. Creating new log.
I20260620 06:06:54.513702 13999 tablet_bootstrap.cc:492] T 0840f1fbda564ebba808038d7e2beb6f P 73042c6c0e8a4affa6e460bb19d7c5c9: No bootstrap required, opened a new log
I20260620 06:06:54.513783 13999 ts_tablet_manager.cc:1397] T 0840f1fbda564ebba808038d7e2beb6f P 73042c6c0e8a4affa6e460bb19d7c5c9: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 06:06:54.514210 13999 raft_consensus.cc:348] T 0840f1fbda564ebba808038d7e2beb6f P 73042c6c0e8a4affa6e460bb19d7c5c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5124d225ee724a998f64980e62a4dd51" member_type: VOTER last_known_addr { host: "127.11.102.1" port: 33471 } } peers { permanent_uuid: "73042c6c0e8a4affa6e460bb19d7c5c9" member_type: VOTER last_known_addr { host: "127.11.102.2" port: 43331 } } peers { permanent_uuid: "e61acac2279e47ffbb9af68a1df4d31d" member_type: VOTER last_known_addr { host: "127.11.102.3" port: 38435 } }
I20260620 06:06:54.514320 13999 raft_consensus.cc:374] T 0840f1fbda564ebba808038d7e2beb6f P 73042c6c0e8a4affa6e460bb19d7c5c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 06:06:54.514367 13999 raft_consensus.cc:729] T 0840f1fbda564ebba808038d7e2beb6f P 73042c6c0e8a4affa6e460bb19d7c5c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73042c6c0e8a4affa6e460bb19d7c5c9, State: Initialized, Role: FOLLOWER
I20260620 06:06:54.514497 13999 consensus_queue.cc:260] T 0840f1fbda564ebba808038d7e2beb6f P 73042c6c0e8a4affa6e460bb19d7c5c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5124d225ee724a998f64980e62a4dd51" member_type: VOTER last_known_addr { host: "127.11.102.1" port: 33471 } } peers { permanent_uuid: "73042c6c0e8a4affa6e460bb19d7c5c9" member_type: VOTER last_known_addr { host: "127.11.102.2" port: 43331 } } peers { permanent_uuid: "e61acac2279e47ffbb9af68a1df4d31d" member_type: VOTER last_known_addr { host: "127.11.102.3" port: 38435 } }
I20260620 06:06:54.514763 13999 ts_tablet_manager.cc:1428] T 0840f1fbda564ebba808038d7e2beb6f P 73042c6c0e8a4affa6e460bb19d7c5c9: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260620 06:06:54.564396 14000 raft_consensus.cc:482] T 0840f1fbda564ebba808038d7e2beb6f P e61acac2279e47ffbb9af68a1df4d31d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 06:06:54.564546 14000 raft_consensus.cc:504] T 0840f1fbda564ebba808038d7e2beb6f P e61acac2279e47ffbb9af68a1df4d31d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5124d225ee724a998f64980e62a4dd51" member_type: VOTER last_known_addr { host: "127.11.102.1" port: 33471 } } peers { permanent_uuid: "73042c6c0e8a4affa6e460bb19d7c5c9" member_type: VOTER last_known_addr { host: "127.11.102.2" port: 43331 } } peers { permanent_uuid: "e61acac2279e47ffbb9af68a1df4d31d" member_type: VOTER last_known_addr { host: "127.11.102.3" port: 38435 } }
I20260620 06:06:54.565056 14000 leader_election.cc:288] T 0840f1fbda564ebba808038d7e2beb6f P e61acac2279e47ffbb9af68a1df4d31d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5124d225ee724a998f64980e62a4dd51 (127.11.102.1:33471), 73042c6c0e8a4affa6e460bb19d7c5c9 (127.11.102.2:43331)
I20260620 06:06:54.565497 13690 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0840f1fbda564ebba808038d7e2beb6f" candidate_uuid: "e61acac2279e47ffbb9af68a1df4d31d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73042c6c0e8a4affa6e460bb19d7c5c9" is_pre_election: true
I20260620 06:06:54.565508 13556 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0840f1fbda564ebba808038d7e2beb6f" candidate_uuid: "e61acac2279e47ffbb9af68a1df4d31d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5124d225ee724a998f64980e62a4dd51" is_pre_election: true
I20260620 06:06:54.565665 13690 raft_consensus.cc:2454] T 0840f1fbda564ebba808038d7e2beb6f P 73042c6c0e8a4affa6e460bb19d7c5c9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e61acac2279e47ffbb9af68a1df4d31d in term 0.
I20260620 06:06:54.565675 13556 raft_consensus.cc:2454] T 0840f1fbda564ebba808038d7e2beb6f P 5124d225ee724a998f64980e62a4dd51 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e61acac2279e47ffbb9af68a1df4d31d in term 0.
I20260620 06:06:54.566067 13768 leader_election.cc:302] T 0840f1fbda564ebba808038d7e2beb6f P e61acac2279e47ffbb9af68a1df4d31d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 73042c6c0e8a4affa6e460bb19d7c5c9, e61acac2279e47ffbb9af68a1df4d31d; no voters:
I20260620 06:06:54.566257 14000 raft_consensus.cc:2781] T 0840f1fbda564ebba808038d7e2beb6f P e61acac2279e47ffbb9af68a1df4d31d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 06:06:54.566318 14000 raft_consensus.cc:482] T 0840f1fbda564ebba808038d7e2beb6f P e61acac2279e47ffbb9af68a1df4d31d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 06:06:54.566358 14000 raft_consensus.cc:3037] T 0840f1fbda564ebba808038d7e2beb6f P e61acac2279e47ffbb9af68a1df4d31d [term 0 FOLLOWER]: Advancing to term 1
I20260620 06:06:54.567528 14000 raft_consensus.cc:504] T 0840f1fbda564ebba808038d7e2beb6f P e61acac2279e47ffbb9af68a1df4d31d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5124d225ee724a998f64980e62a4dd51" member_type: VOTER last_known_addr { host: "127.11.102.1" port: 33471 } } peers { permanent_uuid: "73042c6c0e8a4affa6e460bb19d7c5c9" member_type: VOTER last_known_addr { host: "127.11.102.2" port: 43331 } } peers { permanent_uuid: "e61acac2279e47ffbb9af68a1df4d31d" member_type: VOTER last_known_addr { host: "127.11.102.3" port: 38435 } }
I20260620 06:06:54.567906 14000 leader_election.cc:288] T 0840f1fbda564ebba808038d7e2beb6f P e61acac2279e47ffbb9af68a1df4d31d [CANDIDATE]: Term 1 election: Requested vote from peers 5124d225ee724a998f64980e62a4dd51 (127.11.102.1:33471), 73042c6c0e8a4affa6e460bb19d7c5c9 (127.11.102.2:43331)
I20260620 06:06:54.568195 13556 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0840f1fbda564ebba808038d7e2beb6f" candidate_uuid: "e61acac2279e47ffbb9af68a1df4d31d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5124d225ee724a998f64980e62a4dd51"
I20260620 06:06:54.568329 13556 raft_consensus.cc:3037] T 0840f1fbda564ebba808038d7e2beb6f P 5124d225ee724a998f64980e62a4dd51 [term 0 FOLLOWER]: Advancing to term 1
I20260620 06:06:54.568295 13690 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0840f1fbda564ebba808038d7e2beb6f" candidate_uuid: "e61acac2279e47ffbb9af68a1df4d31d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73042c6c0e8a4affa6e460bb19d7c5c9"
I20260620 06:06:54.568426 13690 raft_consensus.cc:3037] T 0840f1fbda564ebba808038d7e2beb6f P 73042c6c0e8a4affa6e460bb19d7c5c9 [term 0 FOLLOWER]: Advancing to term 1
I20260620 06:06:54.569327 13690 raft_consensus.cc:2454] T 0840f1fbda564ebba808038d7e2beb6f P 73042c6c0e8a4affa6e460bb19d7c5c9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e61acac2279e47ffbb9af68a1df4d31d in term 1.
I20260620 06:06:54.569335 13556 raft_consensus.cc:2454] T 0840f1fbda564ebba808038d7e2beb6f P 5124d225ee724a998f64980e62a4dd51 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e61acac2279e47ffbb9af68a1df4d31d in term 1.
I20260620 06:06:54.569605 13768 leader_election.cc:302] T 0840f1fbda564ebba808038d7e2beb6f P e61acac2279e47ffbb9af68a1df4d31d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 73042c6c0e8a4affa6e460bb19d7c5c9, e61acac2279e47ffbb9af68a1df4d31d; no voters:
I20260620 06:06:54.569782 14000 raft_consensus.cc:2781] T 0840f1fbda564ebba808038d7e2beb6f P e61acac2279e47ffbb9af68a1df4d31d [term 1 FOLLOWER]: Leader election won for term 1
I20260620 06:06:54.569995 14000 raft_consensus.cc:686] T 0840f1fbda564ebba808038d7e2beb6f P e61acac2279e47ffbb9af68a1df4d31d [term 1 LEADER]: Becoming Leader. State: Replica: e61acac2279e47ffbb9af68a1df4d31d, State: Running, Role: LEADER
I20260620 06:06:54.570153 14000 consensus_queue.cc:237] T 0840f1fbda564ebba808038d7e2beb6f P e61acac2279e47ffbb9af68a1df4d31d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5124d225ee724a998f64980e62a4dd51" member_type: VOTER last_known_addr { host: "127.11.102.1" port: 33471 } } peers { permanent_uuid: "73042c6c0e8a4affa6e460bb19d7c5c9" member_type: VOTER last_known_addr { host: "127.11.102.2" port: 43331 } } peers { permanent_uuid: "e61acac2279e47ffbb9af68a1df4d31d" member_type: VOTER last_known_addr { host: "127.11.102.3" port: 38435 } }
I20260620 06:06:54.571785 13399 catalog_manager.cc:5495] T 0840f1fbda564ebba808038d7e2beb6f P e61acac2279e47ffbb9af68a1df4d31d reported cstate change: term changed from 0 to 1, leader changed from <none> to e61acac2279e47ffbb9af68a1df4d31d (127.11.102.3). New cstate: current_term: 1 leader_uuid: "e61acac2279e47ffbb9af68a1df4d31d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5124d225ee724a998f64980e62a4dd51" member_type: VOTER last_known_addr { host: "127.11.102.1" port: 33471 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "73042c6c0e8a4affa6e460bb19d7c5c9" member_type: VOTER last_known_addr { host: "127.11.102.2" port: 43331 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e61acac2279e47ffbb9af68a1df4d31d" member_type: VOTER last_known_addr { host: "127.11.102.3" port: 38435 } health_report { overall_health: HEALTHY } } }
I20260620 06:06:54.593995 14006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:06:54.590945 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39934 (local address 127.11.102.3:38435)
0620 06:06:54.591191 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0620 06:06:54.591198 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 06:06:54.591245 (+ 47us) server_negotiation.cc:407] Connection header received
0620 06:06:54.591291 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:06:54.591294 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:06:54.591342 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:06:54.591434 (+ 92us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 06:06:54.591791 (+ 357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:06:54.592357 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:06:54.593155 (+ 798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:06:54.593366 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:06:54.593441 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:06:54.593647 (+ 206us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 06:06:54.593711 (+ 64us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 06:06:54.593768 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0620 06:06:54.593840 (+ 72us) server_negotiation.cc:299] Negotiation successful
0620 06:06:54.593881 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":181,"thread_start_us":119,"threads_started":1}
I20260620 06:06:54.599843 13690 raft_consensus.cc:1261] T 0840f1fbda564ebba808038d7e2beb6f P 73042c6c0e8a4affa6e460bb19d7c5c9 [term 1 FOLLOWER]: Refusing update from remote peer e61acac2279e47ffbb9af68a1df4d31d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 06:06:54.599941 13556 raft_consensus.cc:1261] T 0840f1fbda564ebba808038d7e2beb6f P 5124d225ee724a998f64980e62a4dd51 [term 1 FOLLOWER]: Refusing update from remote peer e61acac2279e47ffbb9af68a1df4d31d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 06:06:54.600239 14003 consensus_queue.cc:1035] T 0840f1fbda564ebba808038d7e2beb6f P e61acac2279e47ffbb9af68a1df4d31d [LEADER]: Connected to new peer: Peer: permanent_uuid: "73042c6c0e8a4affa6e460bb19d7c5c9" member_type: VOTER last_known_addr { host: "127.11.102.2" port: 43331 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 06:06:54.600381 14000 consensus_queue.cc:1035] T 0840f1fbda564ebba808038d7e2beb6f P e61acac2279e47ffbb9af68a1df4d31d [LEADER]: Connected to new peer: Peer: permanent_uuid: "5124d225ee724a998f64980e62a4dd51" member_type: VOTER last_known_addr { host: "127.11.102.1" port: 33471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 06:06:54.631517 13396 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57842:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 06:06:54.631726 13396 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57842:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 06:06:54.633618 13396 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 794f593770bb429487ad0ec805dae6bf: Sending DeleteTablet for 3 replicas of tablet 0840f1fbda564ebba808038d7e2beb6f
I20260620 06:06:54.634208 13537 tablet_service.cc:1514] Processing DeleteTablet for tablet 0840f1fbda564ebba808038d7e2beb6f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 06:06:54 UTC) from {username='kudu', principal='kudu/127.11.102.62@KRBTEST.COM'} at 127.0.0.1:56016
I20260620 06:06:54.634197 13676 tablet_service.cc:1514] Processing DeleteTablet for tablet 0840f1fbda564ebba808038d7e2beb6f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 06:06:54 UTC) from {username='kudu', principal='kudu/127.11.102.62@KRBTEST.COM'} at 127.0.0.1:50158
I20260620 06:06:54.634208 13817 tablet_service.cc:1514] Processing DeleteTablet for tablet 0840f1fbda564ebba808038d7e2beb6f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 06:06:54 UTC) from {username='kudu', principal='kudu/127.11.102.62@KRBTEST.COM'} at 127.0.0.1:39920
I20260620 06:06:54.634377 13985 tablet_replica.cc:330] T 0840f1fbda564ebba808038d7e2beb6f P 73042c6c0e8a4affa6e460bb19d7c5c9: stopping tablet replica
I20260620 06:06:54.634375 13984 tablet_replica.cc:330] T 0840f1fbda564ebba808038d7e2beb6f P 5124d225ee724a998f64980e62a4dd51: stopping tablet replica
I20260620 06:06:54.634495 13985 raft_consensus.cc:2229] T 0840f1fbda564ebba808038d7e2beb6f P 73042c6c0e8a4affa6e460bb19d7c5c9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 06:06:54.634495 13984 raft_consensus.cc:2229] T 0840f1fbda564ebba808038d7e2beb6f P 5124d225ee724a998f64980e62a4dd51 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 06:06:54.634600 13984 raft_consensus.cc:2258] T 0840f1fbda564ebba808038d7e2beb6f P 5124d225ee724a998f64980e62a4dd51 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 06:06:54.634600 13985 raft_consensus.cc:2258] T 0840f1fbda564ebba808038d7e2beb6f P 73042c6c0e8a4affa6e460bb19d7c5c9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 06:06:54.634766 13986 tablet_replica.cc:330] T 0840f1fbda564ebba808038d7e2beb6f P e61acac2279e47ffbb9af68a1df4d31d: stopping tablet replica
I20260620 06:06:54.634876 13986 raft_consensus.cc:2229] T 0840f1fbda564ebba808038d7e2beb6f P e61acac2279e47ffbb9af68a1df4d31d [term 1 LEADER]: Raft consensus shutting down.
I20260620 06:06:54.635067 13986 raft_consensus.cc:2258] T 0840f1fbda564ebba808038d7e2beb6f P e61acac2279e47ffbb9af68a1df4d31d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 06:06:54.635221 13984 ts_tablet_manager.cc:1905] T 0840f1fbda564ebba808038d7e2beb6f P 5124d225ee724a998f64980e62a4dd51: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 06:06:54.635396 13396 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57842:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260620 06:06:54.635764 13985 ts_tablet_manager.cc:1905] T 0840f1fbda564ebba808038d7e2beb6f P 73042c6c0e8a4affa6e460bb19d7c5c9: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260620 06:06:54.635864 13396 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 06:06:54.637745 13984 ts_tablet_manager.cc:1918] T 0840f1fbda564ebba808038d7e2beb6f P 5124d225ee724a998f64980e62a4dd51: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 06:06:54.637830 13984 log.cc:1196] T 0840f1fbda564ebba808038d7e2beb6f P 5124d225ee724a998f64980e62a4dd51: Deleting WAL directory at /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781935603120653-11672-0/minicluster-data/ts-0/wal/wals/0840f1fbda564ebba808038d7e2beb6f
I20260620 06:06:54.638075 13984 ts_tablet_manager.cc:1939] T 0840f1fbda564ebba808038d7e2beb6f P 5124d225ee724a998f64980e62a4dd51: Deleting consensus metadata
I20260620 06:06:54.638304 13985 ts_tablet_manager.cc:1918] T 0840f1fbda564ebba808038d7e2beb6f P 73042c6c0e8a4affa6e460bb19d7c5c9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 06:06:54.638365 13985 log.cc:1196] T 0840f1fbda564ebba808038d7e2beb6f P 73042c6c0e8a4affa6e460bb19d7c5c9: Deleting WAL directory at /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781935603120653-11672-0/minicluster-data/ts-1/wal/wals/0840f1fbda564ebba808038d7e2beb6f
I20260620 06:06:54.638602 13985 ts_tablet_manager.cc:1939] T 0840f1fbda564ebba808038d7e2beb6f P 73042c6c0e8a4affa6e460bb19d7c5c9: Deleting consensus metadata
I20260620 06:06:54.638921 13387 catalog_manager.cc:4841] TS 5124d225ee724a998f64980e62a4dd51 (127.11.102.1:33471): tablet 0840f1fbda564ebba808038d7e2beb6f (table test-table [id=768e3479107f45278e1aa205078c2447]) successfully deleted
I20260620 06:06:54.639111 13986 ts_tablet_manager.cc:1905] T 0840f1fbda564ebba808038d7e2beb6f P e61acac2279e47ffbb9af68a1df4d31d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 06:06:54.639309 13384 catalog_manager.cc:4841] TS 73042c6c0e8a4affa6e460bb19d7c5c9 (127.11.102.2:43331): tablet 0840f1fbda564ebba808038d7e2beb6f (table test-table [id=768e3479107f45278e1aa205078c2447]) successfully deleted
I20260620 06:06:54.641239 13986 ts_tablet_manager.cc:1918] T 0840f1fbda564ebba808038d7e2beb6f P e61acac2279e47ffbb9af68a1df4d31d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 06:06:54.641340 13986 log.cc:1196] T 0840f1fbda564ebba808038d7e2beb6f P e61acac2279e47ffbb9af68a1df4d31d: Deleting WAL directory at /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781935603120653-11672-0/minicluster-data/ts-2/wal/wals/0840f1fbda564ebba808038d7e2beb6f
I20260620 06:06:54.641644 13986 ts_tablet_manager.cc:1939] T 0840f1fbda564ebba808038d7e2beb6f P e61acac2279e47ffbb9af68a1df4d31d: Deleting consensus metadata
I20260620 06:06:54.642362 13386 catalog_manager.cc:4841] TS e61acac2279e47ffbb9af68a1df4d31d (127.11.102.3:38435): tablet 0840f1fbda564ebba808038d7e2beb6f (table test-table [id=768e3479107f45278e1aa205078c2447]) successfully deleted
I20260620 06:06:54.643244 13537 tablet_service.cc:1467] Processing CreateTablet for tablet 2e346cc2d1254ce5804ba73fb8867a7b (DEFAULT_TABLE table=test-table [id=15529e9565cc48708fe6a504d15d87ff]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 06:06:54.643324 13676 tablet_service.cc:1467] Processing CreateTablet for tablet 2e346cc2d1254ce5804ba73fb8867a7b (DEFAULT_TABLE table=test-table [id=15529e9565cc48708fe6a504d15d87ff]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 06:06:54.643489 13537 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2e346cc2d1254ce5804ba73fb8867a7b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 06:06:54.643476 13817 tablet_service.cc:1467] Processing CreateTablet for tablet 2e346cc2d1254ce5804ba73fb8867a7b (DEFAULT_TABLE table=test-table [id=15529e9565cc48708fe6a504d15d87ff]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 06:06:54.643644 13676 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2e346cc2d1254ce5804ba73fb8867a7b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 06:06:54.643735 13817 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2e346cc2d1254ce5804ba73fb8867a7b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 06:06:54.645537 13999 tablet_bootstrap.cc:492] T 2e346cc2d1254ce5804ba73fb8867a7b P 73042c6c0e8a4affa6e460bb19d7c5c9: Bootstrap starting.
I20260620 06:06:54.645713 13996 tablet_bootstrap.cc:492] T 2e346cc2d1254ce5804ba73fb8867a7b P e61acac2279e47ffbb9af68a1df4d31d: Bootstrap starting.
I20260620 06:06:54.645713 13995 tablet_bootstrap.cc:492] T 2e346cc2d1254ce5804ba73fb8867a7b P 5124d225ee724a998f64980e62a4dd51: Bootstrap starting.
I20260620 06:06:54.646476 13996 tablet_bootstrap.cc:654] T 2e346cc2d1254ce5804ba73fb8867a7b P e61acac2279e47ffbb9af68a1df4d31d: Neither blocks nor log segments found. Creating new log.
I20260620 06:06:54.646488 13995 tablet_bootstrap.cc:654] T 2e346cc2d1254ce5804ba73fb8867a7b P 5124d225ee724a998f64980e62a4dd51: Neither blocks nor log segments found. Creating new log.
I20260620 06:06:54.646525 13999 tablet_bootstrap.cc:654] T 2e346cc2d1254ce5804ba73fb8867a7b P 73042c6c0e8a4affa6e460bb19d7c5c9: Neither blocks nor log segments found. Creating new log.
I20260620 06:06:54.647686 13999 tablet_bootstrap.cc:492] T 2e346cc2d1254ce5804ba73fb8867a7b P 73042c6c0e8a4affa6e460bb19d7c5c9: No bootstrap required, opened a new log
I20260620 06:06:54.647783 13999 ts_tablet_manager.cc:1397] T 2e346cc2d1254ce5804ba73fb8867a7b P 73042c6c0e8a4affa6e460bb19d7c5c9: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260620 06:06:54.647822 13996 tablet_bootstrap.cc:492] T 2e346cc2d1254ce5804ba73fb8867a7b P e61acac2279e47ffbb9af68a1df4d31d: No bootstrap required, opened a new log
I20260620 06:06:54.647858 13995 tablet_bootstrap.cc:492] T 2e346cc2d1254ce5804ba73fb8867a7b P 5124d225ee724a998f64980e62a4dd51: No bootstrap required, opened a new log
I20260620 06:06:54.647902 13996 ts_tablet_manager.cc:1397] T 2e346cc2d1254ce5804ba73fb8867a7b P e61acac2279e47ffbb9af68a1df4d31d: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 06:06:54.647953 13995 ts_tablet_manager.cc:1397] T 2e346cc2d1254ce5804ba73fb8867a7b P 5124d225ee724a998f64980e62a4dd51: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260620 06:06:54.648207 13999 raft_consensus.cc:348] T 2e346cc2d1254ce5804ba73fb8867a7b P 73042c6c0e8a4affa6e460bb19d7c5c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5124d225ee724a998f64980e62a4dd51" member_type: VOTER last_known_addr { host: "127.11.102.1" port: 33471 } } peers { permanent_uuid: "73042c6c0e8a4affa6e460bb19d7c5c9" member_type: VOTER last_known_addr { host: "127.11.102.2" port: 43331 } } peers { permanent_uuid: "e61acac2279e47ffbb9af68a1df4d31d" member_type: VOTER last_known_addr { host: "127.11.102.3" port: 38435 } }
I20260620 06:06:54.648326 13999 raft_consensus.cc:374] T 2e346cc2d1254ce5804ba73fb8867a7b P 73042c6c0e8a4affa6e460bb19d7c5c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 06:06:54.648376 13999 raft_consensus.cc:729] T 2e346cc2d1254ce5804ba73fb8867a7b P 73042c6c0e8a4affa6e460bb19d7c5c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73042c6c0e8a4affa6e460bb19d7c5c9, State: Initialized, Role: FOLLOWER
I20260620 06:06:54.648356 13996 raft_consensus.cc:348] T 2e346cc2d1254ce5804ba73fb8867a7b P e61acac2279e47ffbb9af68a1df4d31d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5124d225ee724a998f64980e62a4dd51" member_type: VOTER last_known_addr { host: "127.11.102.1" port: 33471 } } peers { permanent_uuid: "73042c6c0e8a4affa6e460bb19d7c5c9" member_type: VOTER last_known_addr { host: "127.11.102.2" port: 43331 } } peers { permanent_uuid: "e61acac2279e47ffbb9af68a1df4d31d" member_type: VOTER last_known_addr { host: "127.11.102.3" port: 38435 } }
I20260620 06:06:54.648471 13996 raft_consensus.cc:374] T 2e346cc2d1254ce5804ba73fb8867a7b P e61acac2279e47ffbb9af68a1df4d31d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 06:06:54.648407 13995 raft_consensus.cc:348] T 2e346cc2d1254ce5804ba73fb8867a7b P 5124d225ee724a998f64980e62a4dd51 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5124d225ee724a998f64980e62a4dd51" member_type: VOTER last_known_addr { host: "127.11.102.1" port: 33471 } } peers { permanent_uuid: "73042c6c0e8a4affa6e460bb19d7c5c9" member_type: VOTER last_known_addr { host: "127.11.102.2" port: 43331 } } peers { permanent_uuid: "e61acac2279e47ffbb9af68a1df4d31d" member_type: VOTER last_known_addr { host: "127.11.102.3" port: 38435 } }
I20260620 06:06:54.648522 13996 raft_consensus.cc:729] T 2e346cc2d1254ce5804ba73fb8867a7b P e61acac2279e47ffbb9af68a1df4d31d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e61acac2279e47ffbb9af68a1df4d31d, State: Initialized, Role: FOLLOWER
I20260620 06:06:54.648530 13995 raft_consensus.cc:374] T 2e346cc2d1254ce5804ba73fb8867a7b P 5124d225ee724a998f64980e62a4dd51 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 06:06:54.648569 13995 raft_consensus.cc:729] T 2e346cc2d1254ce5804ba73fb8867a7b P 5124d225ee724a998f64980e62a4dd51 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5124d225ee724a998f64980e62a4dd51, State: Initialized, Role: FOLLOWER
I20260620 06:06:54.648530 13999 consensus_queue.cc:260] T 2e346cc2d1254ce5804ba73fb8867a7b P 73042c6c0e8a4affa6e460bb19d7c5c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5124d225ee724a998f64980e62a4dd51" member_type: VOTER last_known_addr { host: "127.11.102.1" port: 33471 } } peers { permanent_uuid: "73042c6c0e8a4affa6e460bb19d7c5c9" member_type: VOTER last_known_addr { host: "127.11.102.2" port: 43331 } } peers { permanent_uuid: "e61acac2279e47ffbb9af68a1df4d31d" member_type: VOTER last_known_addr { host: "127.11.102.3" port: 38435 } }
I20260620 06:06:54.648648 13996 consensus_queue.cc:260] T 2e346cc2d1254ce5804ba73fb8867a7b P e61acac2279e47ffbb9af68a1df4d31d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5124d225ee724a998f64980e62a4dd51" member_type: VOTER last_known_addr { host: "127.11.102.1" port: 33471 } } peers { permanent_uuid: "73042c6c0e8a4affa6e460bb19d7c5c9" member_type: VOTER last_known_addr { host: "127.11.102.2" port: 43331 } } peers { permanent_uuid: "e61acac2279e47ffbb9af68a1df4d31d" member_type: VOTER last_known_addr { host: "127.11.102.3" port: 38435 } }
I20260620 06:06:54.648685 13995 consensus_queue.cc:260] T 2e346cc2d1254ce5804ba73fb8867a7b P 5124d225ee724a998f64980e62a4dd51 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5124d225ee724a998f64980e62a4dd51" member_type: VOTER last_known_addr { host: "127.11.102.1" port: 33471 } } peers { permanent_uuid: "73042c6c0e8a4affa6e460bb19d7c5c9" member_type: VOTER last_known_addr { host: "127.11.102.2" port: 43331 } } peers { permanent_uuid: "e61acac2279e47ffbb9af68a1df4d31d" member_type: VOTER last_known_addr { host: "127.11.102.3" port: 38435 } }
I20260620 06:06:54.648823 13999 ts_tablet_manager.cc:1428] T 2e346cc2d1254ce5804ba73fb8867a7b P 73042c6c0e8a4affa6e460bb19d7c5c9: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260620 06:06:54.648910 13996 ts_tablet_manager.cc:1428] T 2e346cc2d1254ce5804ba73fb8867a7b P e61acac2279e47ffbb9af68a1df4d31d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260620 06:06:54.648988 13995 ts_tablet_manager.cc:1428] T 2e346cc2d1254ce5804ba73fb8867a7b P 5124d225ee724a998f64980e62a4dd51: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260620 06:06:54.903697 14010 raft_consensus.cc:482] T 2e346cc2d1254ce5804ba73fb8867a7b P e61acac2279e47ffbb9af68a1df4d31d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 06:06:54.903856 14010 raft_consensus.cc:504] T 2e346cc2d1254ce5804ba73fb8867a7b P e61acac2279e47ffbb9af68a1df4d31d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5124d225ee724a998f64980e62a4dd51" member_type: VOTER last_known_addr { host: "127.11.102.1" port: 33471 } } peers { permanent_uuid: "73042c6c0e8a4affa6e460bb19d7c5c9" member_type: VOTER last_known_addr { host: "127.11.102.2" port: 43331 } } peers { permanent_uuid: "e61acac2279e47ffbb9af68a1df4d31d" member_type: VOTER last_known_addr { host: "127.11.102.3" port: 38435 } }
I20260620 06:06:54.904321 14010 leader_election.cc:288] T 2e346cc2d1254ce5804ba73fb8867a7b P e61acac2279e47ffbb9af68a1df4d31d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5124d225ee724a998f64980e62a4dd51 (127.11.102.1:33471), 73042c6c0e8a4affa6e460bb19d7c5c9 (127.11.102.2:43331)
I20260620 06:06:54.904822 13557 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e346cc2d1254ce5804ba73fb8867a7b" candidate_uuid: "e61acac2279e47ffbb9af68a1df4d31d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5124d225ee724a998f64980e62a4dd51" is_pre_election: true
I20260620 06:06:54.904824 13690 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e346cc2d1254ce5804ba73fb8867a7b" candidate_uuid: "e61acac2279e47ffbb9af68a1df4d31d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73042c6c0e8a4affa6e460bb19d7c5c9" is_pre_election: true
I20260620 06:06:54.904968 13557 raft_consensus.cc:2454] T 2e346cc2d1254ce5804ba73fb8867a7b P 5124d225ee724a998f64980e62a4dd51 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e61acac2279e47ffbb9af68a1df4d31d in term 0.
I20260620 06:06:54.905009 13690 raft_consensus.cc:2454] T 2e346cc2d1254ce5804ba73fb8867a7b P 73042c6c0e8a4affa6e460bb19d7c5c9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e61acac2279e47ffbb9af68a1df4d31d in term 0.
I20260620 06:06:54.905331 13771 leader_election.cc:302] T 2e346cc2d1254ce5804ba73fb8867a7b P e61acac2279e47ffbb9af68a1df4d31d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5124d225ee724a998f64980e62a4dd51, e61acac2279e47ffbb9af68a1df4d31d; no voters:
I20260620 06:06:54.905544 14010 raft_consensus.cc:2781] T 2e346cc2d1254ce5804ba73fb8867a7b P e61acac2279e47ffbb9af68a1df4d31d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 06:06:54.905609 14010 raft_consensus.cc:482] T 2e346cc2d1254ce5804ba73fb8867a7b P e61acac2279e47ffbb9af68a1df4d31d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 06:06:54.905638 14010 raft_consensus.cc:3037] T 2e346cc2d1254ce5804ba73fb8867a7b P e61acac2279e47ffbb9af68a1df4d31d [term 0 FOLLOWER]: Advancing to term 1
I20260620 06:06:54.906564 14010 raft_consensus.cc:504] T 2e346cc2d1254ce5804ba73fb8867a7b P e61acac2279e47ffbb9af68a1df4d31d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5124d225ee724a998f64980e62a4dd51" member_type: VOTER last_known_addr { host: "127.11.102.1" port: 33471 } } peers { permanent_uuid: "73042c6c0e8a4affa6e460bb19d7c5c9" member_type: VOTER last_known_addr { host: "127.11.102.2" port: 43331 } } peers { permanent_uuid: "e61acac2279e47ffbb9af68a1df4d31d" member_type: VOTER last_known_addr { host: "127.11.102.3" port: 38435 } }
I20260620 06:06:54.906904 14010 leader_election.cc:288] T 2e346cc2d1254ce5804ba73fb8867a7b P e61acac2279e47ffbb9af68a1df4d31d [CANDIDATE]: Term 1 election: Requested vote from peers 5124d225ee724a998f64980e62a4dd51 (127.11.102.1:33471), 73042c6c0e8a4affa6e460bb19d7c5c9 (127.11.102.2:43331)
I20260620 06:06:54.907238 13557 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e346cc2d1254ce5804ba73fb8867a7b" candidate_uuid: "e61acac2279e47ffbb9af68a1df4d31d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5124d225ee724a998f64980e62a4dd51"
I20260620 06:06:54.907339 13557 raft_consensus.cc:3037] T 2e346cc2d1254ce5804ba73fb8867a7b P 5124d225ee724a998f64980e62a4dd51 [term 0 FOLLOWER]: Advancing to term 1
I20260620 06:06:54.907279 13690 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e346cc2d1254ce5804ba73fb8867a7b" candidate_uuid: "e61acac2279e47ffbb9af68a1df4d31d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73042c6c0e8a4affa6e460bb19d7c5c9"
I20260620 06:06:54.907413 13690 raft_consensus.cc:3037] T 2e346cc2d1254ce5804ba73fb8867a7b P 73042c6c0e8a4affa6e460bb19d7c5c9 [term 0 FOLLOWER]: Advancing to term 1
I20260620 06:06:54.908226 13557 raft_consensus.cc:2454] T 2e346cc2d1254ce5804ba73fb8867a7b P 5124d225ee724a998f64980e62a4dd51 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e61acac2279e47ffbb9af68a1df4d31d in term 1.
I20260620 06:06:54.908511 13771 leader_election.cc:302] T 2e346cc2d1254ce5804ba73fb8867a7b P e61acac2279e47ffbb9af68a1df4d31d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5124d225ee724a998f64980e62a4dd51, e61acac2279e47ffbb9af68a1df4d31d; no voters:
I20260620 06:06:54.908540 13690 raft_consensus.cc:2454] T 2e346cc2d1254ce5804ba73fb8867a7b P 73042c6c0e8a4affa6e460bb19d7c5c9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e61acac2279e47ffbb9af68a1df4d31d in term 1.
I20260620 06:06:54.908668 14010 raft_consensus.cc:2781] T 2e346cc2d1254ce5804ba73fb8867a7b P e61acac2279e47ffbb9af68a1df4d31d [term 1 FOLLOWER]: Leader election won for term 1
I20260620 06:06:54.908835 14010 raft_consensus.cc:686] T 2e346cc2d1254ce5804ba73fb8867a7b P e61acac2279e47ffbb9af68a1df4d31d [term 1 LEADER]: Becoming Leader. State: Replica: e61acac2279e47ffbb9af68a1df4d31d, State: Running, Role: LEADER
I20260620 06:06:54.908983 14010 consensus_queue.cc:237] T 2e346cc2d1254ce5804ba73fb8867a7b P e61acac2279e47ffbb9af68a1df4d31d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5124d225ee724a998f64980e62a4dd51" member_type: VOTER last_known_addr { host: "127.11.102.1" port: 33471 } } peers { permanent_uuid: "73042c6c0e8a4affa6e460bb19d7c5c9" member_type: VOTER last_known_addr { host: "127.11.102.2" port: 43331 } } peers { permanent_uuid: "e61acac2279e47ffbb9af68a1df4d31d" member_type: VOTER last_known_addr { host: "127.11.102.3" port: 38435 } }
I20260620 06:06:54.910575 13399 catalog_manager.cc:5495] T 2e346cc2d1254ce5804ba73fb8867a7b P e61acac2279e47ffbb9af68a1df4d31d reported cstate change: term changed from 0 to 1, leader changed from <none> to e61acac2279e47ffbb9af68a1df4d31d (127.11.102.3). New cstate: current_term: 1 leader_uuid: "e61acac2279e47ffbb9af68a1df4d31d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5124d225ee724a998f64980e62a4dd51" member_type: VOTER last_known_addr { host: "127.11.102.1" port: 33471 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "73042c6c0e8a4affa6e460bb19d7c5c9" member_type: VOTER last_known_addr { host: "127.11.102.2" port: 43331 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e61acac2279e47ffbb9af68a1df4d31d" member_type: VOTER last_known_addr { host: "127.11.102.3" port: 38435 } health_report { overall_health: HEALTHY } } }
I20260620 06:06:54.949432 13690 raft_consensus.cc:1261] T 2e346cc2d1254ce5804ba73fb8867a7b P 73042c6c0e8a4affa6e460bb19d7c5c9 [term 1 FOLLOWER]: Refusing update from remote peer e61acac2279e47ffbb9af68a1df4d31d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 06:06:54.949486 13557 raft_consensus.cc:1261] T 2e346cc2d1254ce5804ba73fb8867a7b P 5124d225ee724a998f64980e62a4dd51 [term 1 FOLLOWER]: Refusing update from remote peer e61acac2279e47ffbb9af68a1df4d31d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 06:06:54.949898 14003 consensus_queue.cc:1035] T 2e346cc2d1254ce5804ba73fb8867a7b P e61acac2279e47ffbb9af68a1df4d31d [LEADER]: Connected to new peer: Peer: permanent_uuid: "5124d225ee724a998f64980e62a4dd51" member_type: VOTER last_known_addr { host: "127.11.102.1" port: 33471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 06:06:54.950065 14010 consensus_queue.cc:1035] T 2e346cc2d1254ce5804ba73fb8867a7b P e61acac2279e47ffbb9af68a1df4d31d [LEADER]: Connected to new peer: Peer: permanent_uuid: "73042c6c0e8a4affa6e460bb19d7c5c9" member_type: VOTER last_known_addr { host: "127.11.102.2" port: 43331 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260620 06:06:54.963049 13968 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260620 06:06:56.511798 13399 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57842:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 06:06:56.511953 13399 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57842:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 06:06:56.514226 13399 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 794f593770bb429487ad0ec805dae6bf: Sending DeleteTablet for 3 replicas of tablet 2e346cc2d1254ce5804ba73fb8867a7b
I20260620 06:06:56.514886 13537 tablet_service.cc:1514] Processing DeleteTablet for tablet 2e346cc2d1254ce5804ba73fb8867a7b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 06:06:56 UTC) from {username='kudu', principal='kudu/127.11.102.62@KRBTEST.COM'} at 127.0.0.1:56016
I20260620 06:06:56.514884 13817 tablet_service.cc:1514] Processing DeleteTablet for tablet 2e346cc2d1254ce5804ba73fb8867a7b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 06:06:56 UTC) from {username='kudu', principal='kudu/127.11.102.62@KRBTEST.COM'} at 127.0.0.1:39920
I20260620 06:06:56.514889 13676 tablet_service.cc:1514] Processing DeleteTablet for tablet 2e346cc2d1254ce5804ba73fb8867a7b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 06:06:56 UTC) from {username='kudu', principal='kudu/127.11.102.62@KRBTEST.COM'} at 127.0.0.1:50158
I20260620 06:06:56.515235 14043 tablet_replica.cc:330] T 2e346cc2d1254ce5804ba73fb8867a7b P 5124d225ee724a998f64980e62a4dd51: stopping tablet replica
I20260620 06:06:56.515260 14044 tablet_replica.cc:330] T 2e346cc2d1254ce5804ba73fb8867a7b P 73042c6c0e8a4affa6e460bb19d7c5c9: stopping tablet replica
I20260620 06:06:56.515305 14042 tablet_replica.cc:330] T 2e346cc2d1254ce5804ba73fb8867a7b P e61acac2279e47ffbb9af68a1df4d31d: stopping tablet replica
I20260620 06:06:56.515415 14044 raft_consensus.cc:2229] T 2e346cc2d1254ce5804ba73fb8867a7b P 73042c6c0e8a4affa6e460bb19d7c5c9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 06:06:56.515416 14043 raft_consensus.cc:2229] T 2e346cc2d1254ce5804ba73fb8867a7b P 5124d225ee724a998f64980e62a4dd51 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 06:06:56.515462 14042 raft_consensus.cc:2229] T 2e346cc2d1254ce5804ba73fb8867a7b P e61acac2279e47ffbb9af68a1df4d31d [term 1 LEADER]: Raft consensus shutting down.
I20260620 06:06:56.515558 11672 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9HMp43/build/debug/bin/kudu with pid 13472
I20260620 06:06:56.516994 14042 raft_consensus.cc:2258] T 2e346cc2d1254ce5804ba73fb8867a7b P e61acac2279e47ffbb9af68a1df4d31d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 06:06:56.517247 14044 raft_consensus.cc:2258] T 2e346cc2d1254ce5804ba73fb8867a7b P 73042c6c0e8a4affa6e460bb19d7c5c9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 06:06:56.517635 14042 ts_tablet_manager.cc:1905] T 2e346cc2d1254ce5804ba73fb8867a7b P e61acac2279e47ffbb9af68a1df4d31d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 06:06:56.517792 14044 ts_tablet_manager.cc:1905] T 2e346cc2d1254ce5804ba73fb8867a7b P 73042c6c0e8a4affa6e460bb19d7c5c9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 06:06:56.520205 14044 ts_tablet_manager.cc:1918] T 2e346cc2d1254ce5804ba73fb8867a7b P 73042c6c0e8a4affa6e460bb19d7c5c9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260620 06:06:56.520205 14042 ts_tablet_manager.cc:1918] T 2e346cc2d1254ce5804ba73fb8867a7b P e61acac2279e47ffbb9af68a1df4d31d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260620 06:06:56.520283 14042 log.cc:1196] T 2e346cc2d1254ce5804ba73fb8867a7b P e61acac2279e47ffbb9af68a1df4d31d: Deleting WAL directory at /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781935603120653-11672-0/minicluster-data/ts-2/wal/wals/2e346cc2d1254ce5804ba73fb8867a7b
I20260620 06:06:56.520283 14044 log.cc:1196] T 2e346cc2d1254ce5804ba73fb8867a7b P 73042c6c0e8a4affa6e460bb19d7c5c9: Deleting WAL directory at /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781935603120653-11672-0/minicluster-data/ts-1/wal/wals/2e346cc2d1254ce5804ba73fb8867a7b
I20260620 06:06:56.520561 14044 ts_tablet_manager.cc:1939] T 2e346cc2d1254ce5804ba73fb8867a7b P 73042c6c0e8a4affa6e460bb19d7c5c9: Deleting consensus metadata
I20260620 06:06:56.520578 14042 ts_tablet_manager.cc:1939] T 2e346cc2d1254ce5804ba73fb8867a7b P e61acac2279e47ffbb9af68a1df4d31d: Deleting consensus metadata
I20260620 06:06:56.521166 13384 catalog_manager.cc:4841] TS 73042c6c0e8a4affa6e460bb19d7c5c9 (127.11.102.2:43331): tablet 2e346cc2d1254ce5804ba73fb8867a7b (table test-table [id=15529e9565cc48708fe6a504d15d87ff]) successfully deleted
I20260620 06:06:56.521235 13386 catalog_manager.cc:4841] TS e61acac2279e47ffbb9af68a1df4d31d (127.11.102.3:38435): tablet 2e346cc2d1254ce5804ba73fb8867a7b (table test-table [id=15529e9565cc48708fe6a504d15d87ff]) successfully deleted
W20260620 06:06:56.522375 13387 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.11.102.1:33471 (error 108)
I20260620 06:06:56.523226 11672 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9HMp43/build/debug/bin/kudu with pid 13612
I20260620 06:06:56.524026 14046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:06:56.523036 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.102.1:33471 (local address 127.0.0.1:56024)
0620 06:06:56.523294 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0620 06:06:56.523885 (+ 591us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.102.1:33471: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":108,"threads_started":1}
W20260620 06:06:56.524711 13387 catalog_manager.cc:4584] TS 5124d225ee724a998f64980e62a4dd51 (127.11.102.1:33471): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 2e346cc2d1254ce5804ba73fb8867a7b: Network error: Client connection negotiation failed: client connection to 127.11.102.1:33471: connect: Connection refused (error 111)
I20260620 06:06:56.530439 11672 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9HMp43/build/debug/bin/kudu with pid 13752
I20260620 06:06:56.536752 11672 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9HMp43/build/debug/bin/kudu with pid 13368
2026-06-20T06:06:56Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (6268 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781935603120653-11672-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 06:06:56 dist-test-slave-9hm5 krb5kdc[14049](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 06:06:56 dist-test-slave-9hm5 krb5kdc[14049](info): set up 2 sockets
Jun 20 06:06:56 dist-test-slave-9hm5 krb5kdc[14049](info): commencing operation
krb5kdc: starting...
W20260620 06:06:57.575738 11672 mini_kdc.cc:121] Time spent starting KDC: real 1.016s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781935603120653-11672-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781935603120653-11672-0/krb5kdc/test-user.keytab.
Jun 20 06:06:57 dist-test-slave-9hm5 krb5kdc[14049](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935617, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T06:06:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T06:06:57Z Disabled control of system clock
WARNING: no policy specified for kudu/127.11.102.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.102.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.102.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781935603120653-11672-0/krb5kdc/kudu_127.11.102.62.keytab.
Entry for principal kudu/127.11.102.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781935603120653-11672-0/krb5kdc/kudu_127.11.102.62.keytab.
I20260620 06:06:57.694522 11672 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9HMp43/build/debug/bin/kudu
/tmp/dist-test-task9HMp43/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781935603120653-11672-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781935603120653-11672-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781935603120653-11672-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781935603120653-11672-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.102.62:35025
--webserver_interface=127.11.102.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781935603120653-11672-0/krb5kdc/kudu_127.11.102.62.keytab
--principal=kudu/127.11.102.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.102.20:36841
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.102.62:35025
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781935603120653-11672-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781935603120653-11672-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781935603120653-11672-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 06:06:57.802156 14062 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:06:57.802417 14062 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:06:57.802456 14062 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:06:57.806370 14062 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 06:06:57.806428 14062 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:06:57.806447 14062 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:06:57.806464 14062 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 06:06:57.806505 14062 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 06:06:57.811820 14062 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.102.20:36841
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781935603120653-11672-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781935603120653-11672-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.102.62:35025
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781935603120653-11672-0/krb5kdc/kudu_127.11.102.62.keytab
--principal=kudu/127.11.102.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.102.62:35025
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781935603120653-11672-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.102.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14062
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781935603120653-11672-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 05:28:48 UTC on ccycloud.quasar-fixhms.root.comops.site
build id 4457020
I20260620 06:06:57.812812 14062 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:06:57.813812 14062 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 06:06:57.822134 14070 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:06:57.822139 14068 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:06:57.822293 14062 server_base.cc:1034] running on GCE node
W20260620 06:06:57.822543 14067 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:06:57.823112 14062 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:06:57.824193 14062 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:06:57.825347 14062 hybrid_clock.cc:648] HybridClock initialized: now 1781935617825344 us; error 37 us; skew 500 ppm
Jun 20 06:06:57 dist-test-slave-9hm5 krb5kdc[14049](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935617, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.102.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 06:06:57.827975 14062 init.cc:377] Logged in from keytab as kudu/127.11.102.62@KRBTEST.COM (short username kudu)
I20260620 06:06:57.828774 14062 webserver.cc:460] Webserver started at http://127.11.102.62:43777/ using document root <none> and password file <none>
I20260620 06:06:57.829309 14062 fs_manager.cc:359] Metadata directory not provided
I20260620 06:06:57.829367 14062 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:06:57.829591 14062 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 06:06:57.831514 14062 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781935603120653-11672-0/minicluster-data/master-0/data/instance:
uuid: "581478f63a8e451492da81f82c941019"
format_stamp: "Formatted at 2026-06-20 06:06:57 on dist-test-slave-9hm5"
server_key: "ca1c0768e73357bee888a631c067ad59"
server_key_iv: "d39569ece251f8c40cf704c8b9b26775"
server_key_version: "encryptionkey@0"
I20260620 06:06:57.832120 14062 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781935603120653-11672-0/minicluster-data/master-0/wal/instance:
uuid: "581478f63a8e451492da81f82c941019"
format_stamp: "Formatted at 2026-06-20 06:06:57 on dist-test-slave-9hm5"
server_key: "ca1c0768e73357bee888a631c067ad59"
server_key_iv: "d39569ece251f8c40cf704c8b9b26775"
server_key_version: "encryptionkey@0"
I20260620 06:06:57.835542 14062 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260620 06:06:57.837908 14077 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:06:57.839207 14062 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 06:06:57.839327 14062 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781935603120653-11672-0/minicluster-data/master-0/data,/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781935603120653-11672-0/minicluster-data/master-0/wal
uuid: "581478f63a8e451492da81f82c941019"
format_stamp: "Formatted at 2026-06-20 06:06:57 on dist-test-slave-9hm5"
server_key: "ca1c0768e73357bee888a631c067ad59"
server_key_iv: "d39569ece251f8c40cf704c8b9b26775"
server_key_version: "encryptionkey@0"
I20260620 06:06:57.839429 14062 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781935603120653-11672-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781935603120653-11672-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781935603120653-11672-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:06:57.861428 14062 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:06:57.864650 14062 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:06:57.864857 14062 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:06:57.871814 14062 rpc_server.cc:307] RPC server started. Bound to: 127.11.102.62:35025
I20260620 06:06:57.871893 14129 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.102.62:35025 every 8 connection(s)
I20260620 06:06:57.872802 14062 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781935603120653-11672-0/minicluster-data/master-0/data/info.pb
I20260620 06:06:57.875582 14130 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 06:06:57.880669 11672 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9HMp43/build/debug/bin/kudu as pid 14062
I20260620 06:06:57.880781 11672 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781935603120653-11672-0/minicluster-data/master-0/wal/instance
I20260620 06:06:57.881045 11672 external_mini_cluster.cc:1428] Setting key e0362d42cd197d94c2a28c1bea4d8773
I20260620 06:06:57.881301 14130 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 581478f63a8e451492da81f82c941019: Bootstrap starting.
I20260620 06:06:57.883931 14130 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 581478f63a8e451492da81f82c941019: Neither blocks nor log segments found. Creating new log.
I20260620 06:06:57.884770 14130 log.cc:824] T 00000000000000000000000000000000 P 581478f63a8e451492da81f82c941019: Log is configured to *not* fsync() on all Append() calls
I20260620 06:06:57.886857 14130 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 581478f63a8e451492da81f82c941019: No bootstrap required, opened a new log
Jun 20 06:06:57 dist-test-slave-9hm5 krb5kdc[14049](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935617, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.102.62@KRBTEST.COM
I20260620 06:06:57.890919 14130 raft_consensus.cc:348] T 00000000000000000000000000000000 P 581478f63a8e451492da81f82c941019 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "581478f63a8e451492da81f82c941019" member_type: VOTER last_known_addr { host: "127.11.102.62" port: 35025 } }
I20260620 06:06:57.891132 14130 raft_consensus.cc:374] T 00000000000000000000000000000000 P 581478f63a8e451492da81f82c941019 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 06:06:57.891189 14130 raft_consensus.cc:729] T 00000000000000000000000000000000 P 581478f63a8e451492da81f82c941019 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 581478f63a8e451492da81f82c941019, State: Initialized, Role: FOLLOWER
I20260620 06:06:57.891680 14130 consensus_queue.cc:260] T 00000000000000000000000000000000 P 581478f63a8e451492da81f82c941019 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "581478f63a8e451492da81f82c941019" member_type: VOTER last_known_addr { host: "127.11.102.62" port: 35025 } }
I20260620 06:06:57.891805 14130 raft_consensus.cc:388] T 00000000000000000000000000000000 P 581478f63a8e451492da81f82c941019 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 06:06:57.891886 14130 raft_consensus.cc:482] T 00000000000000000000000000000000 P 581478f63a8e451492da81f82c941019 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 06:06:57.891990 14130 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 581478f63a8e451492da81f82c941019 [term 0 FOLLOWER]: Advancing to term 1
I20260620 06:06:57.892809 14130 raft_consensus.cc:504] T 00000000000000000000000000000000 P 581478f63a8e451492da81f82c941019 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "581478f63a8e451492da81f82c941019" member_type: VOTER last_known_addr { host: "127.11.102.62" port: 35025 } }
I20260620 06:06:57.893186 14130 leader_election.cc:302] T 00000000000000000000000000000000 P 581478f63a8e451492da81f82c941019 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 581478f63a8e451492da81f82c941019; no voters:
I20260620 06:06:57.893565 14130 leader_election.cc:288] T 00000000000000000000000000000000 P 581478f63a8e451492da81f82c941019 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 06:06:57.893666 14135 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 581478f63a8e451492da81f82c941019 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 06:06:57.893898 14135 raft_consensus.cc:686] T 00000000000000000000000000000000 P 581478f63a8e451492da81f82c941019 [term 1 LEADER]: Becoming Leader. State: Replica: 581478f63a8e451492da81f82c941019, State: Running, Role: LEADER
I20260620 06:06:57.894238 14135 consensus_queue.cc:237] T 00000000000000000000000000000000 P 581478f63a8e451492da81f82c941019 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "581478f63a8e451492da81f82c941019" member_type: VOTER last_known_addr { host: "127.11.102.62" port: 35025 } }
I20260620 06:06:57.894785 14130 sys_catalog.cc:564] T 00000000000000000000000000000000 P 581478f63a8e451492da81f82c941019 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 06:06:57.895479 14135 sys_catalog.cc:455] T 00000000000000000000000000000000 P 581478f63a8e451492da81f82c941019 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 581478f63a8e451492da81f82c941019. Latest consensus state: current_term: 1 leader_uuid: "581478f63a8e451492da81f82c941019" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "581478f63a8e451492da81f82c941019" member_type: VOTER last_known_addr { host: "127.11.102.62" port: 35025 } } }
I20260620 06:06:57.895684 14135 sys_catalog.cc:458] T 00000000000000000000000000000000 P 581478f63a8e451492da81f82c941019 [sys.catalog]: This master's current role is: LEADER
I20260620 06:06:57.895942 14136 sys_catalog.cc:455] T 00000000000000000000000000000000 P 581478f63a8e451492da81f82c941019 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "581478f63a8e451492da81f82c941019" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "581478f63a8e451492da81f82c941019" member_type: VOTER last_known_addr { host: "127.11.102.62" port: 35025 } } }
I20260620 06:06:57.896042 14136 sys_catalog.cc:458] T 00000000000000000000000000000000 P 581478f63a8e451492da81f82c941019 [sys.catalog]: This master's current role is: LEADER
I20260620 06:06:57.896071 14133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:06:57.882414 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41794 (local address 127.11.102.62:35025)
0620 06:06:57.882819 (+ 405us) server_negotiation.cc:206] Beginning negotiation
0620 06:06:57.882837 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0620 06:06:57.882879 (+ 42us) server_negotiation.cc:407] Connection header received
0620 06:06:57.883699 (+ 820us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:06:57.883721 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:06:57.884143 (+ 422us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:06:57.884511 (+ 368us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:06:57.885235 (+ 724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:06:57.886596 (+ 1361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:06:57.887480 (+ 884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:06:57.888173 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:06:57.888332 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:06:57.890960 (+ 2628us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:06:57.890965 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:06:57.890977 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:06:57.891002 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:06:57.892798 (+ 1796us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:06:57.893252 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:06:57.893256 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:06:57.893260 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:06:57.893434 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:06:57.893750 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:06:57.893755 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:06:57.893757 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:06:57.894279 (+ 522us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:06:57.894465 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0620 06:06:57.894765 (+ 300us) server_negotiation.cc:299] Negotiation successful
0620 06:06:57.895051 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":249,"thread_start_us":97,"threads_started":1}
I20260620 06:06:57.896624 14143 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 06:06:57.899814 14143 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 06:06:57.905071 14143 catalog_manager.cc:1347] Generated new cluster ID: e92a40f272134038a191016742e2c7b7
I20260620 06:06:57.905140 14143 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 06:06:57.914031 14143 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 06:06:57.914880 14143 catalog_manager.cc:1504] Loading token signing keys...
I20260620 06:06:57.920876 14143 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 581478f63a8e451492da81f82c941019: Generated new TSK 0
I20260620 06:06:57.921686 14143 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.11.102.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.102.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.102.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781935603120653-11672-0/krb5kdc/kudu_127.11.102.1.keytab.
Entry for principal kudu/127.11.102.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781935603120653-11672-0/krb5kdc/kudu_127.11.102.1.keytab.
I20260620 06:06:57.954749 11672 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9HMp43/build/debug/bin/kudu
/tmp/dist-test-task9HMp43/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781935603120653-11672-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781935603120653-11672-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781935603120653-11672-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781935603120653-11672-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.102.1:0
--local_ip_for_outbound_sockets=127.11.102.1
--webserver_interface=127.11.102.1
--webserver_port=0
--tserver_master_addrs=127.11.102.62:35025
--keytab_file=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781935603120653-11672-0/krb5kdc/kudu_127.11.102.1.keytab
--principal=kudu/127.11.102.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.102.20:36841
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781935603120653-11672-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781935603120653-11672-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781935603120653-11672-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 06:06:58.060812 14156 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:06:58.061072 14156 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:06:58.061146 14156 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:06:58.065057 14156 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:06:58.065129 14156 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:06:58.065220 14156 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.102.1
I20260620 06:06:58.070412 14156 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.102.20:36841
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781935603120653-11672-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781935603120653-11672-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781935603120653-11672-0/krb5kdc/kudu_127.11.102.1.keytab
--principal=kudu/127.11.102.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.102.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781935603120653-11672-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.11.102.1
--webserver_port=0
--tserver_master_addrs=127.11.102.62:35025
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14156
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.102.1
--log_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781935603120653-11672-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 05:28:48 UTC on ccycloud.quasar-fixhms.root.comops.site
build id 4457020
I20260620 06:06:58.071365 14156 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:06:58.072363 14156 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 06:06:58.081053 14162 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:06:58.081027 14164 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:06:58.081218 14161 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:06:58.081223 14156 server_base.cc:1034] running on GCE node
I20260620 06:06:58.081751 14156 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:06:58.082336 14156 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:06:58.083514 14156 hybrid_clock.cc:648] HybridClock initialized: now 1781935618083502 us; error 27 us; skew 500 ppm
Jun 20 06:06:58 dist-test-slave-9hm5 krb5kdc[14049](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935618, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.102.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 06:06:58.086112 14156 init.cc:377] Logged in from keytab as kudu/127.11.102.1@KRBTEST.COM (short username kudu)
I20260620 06:06:58.087164 14156 webserver.cc:460] Webserver started at http://127.11.102.1:37423/ using document root <none> and password file <none>
I20260620 06:06:58.087826 14156 fs_manager.cc:359] Metadata directory not provided
I20260620 06:06:58.087888 14156 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:06:58.088068 14156 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 06:06:58.089915 14156 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781935603120653-11672-0/minicluster-data/ts-0/data/instance:
uuid: "5ccd0d2539544400894bf073ec9a8538"
format_stamp: "Formatted at 2026-06-20 06:06:58 on dist-test-slave-9hm5"
server_key: "4d088aeb1b3791112e609dcc7d841538"
server_key_iv: "bdf43970c44b0d892ee3d4f265dd8520"
server_key_version: "encryptionkey@0"
I20260620 06:06:58.090394 14156 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781935603120653-11672-0/minicluster-data/ts-0/wal/instance:
uuid: "5ccd0d2539544400894bf073ec9a8538"
format_stamp: "Formatted at 2026-06-20 06:06:58 on dist-test-slave-9hm5"
server_key: "4d088aeb1b3791112e609dcc7d841538"
server_key_iv: "bdf43970c44b0d892ee3d4f265dd8520"
server_key_version: "encryptionkey@0"
I20260620 06:06:58.094081 14156 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260620 06:06:58.096468 14171 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:06:58.097654 14156 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 06:06:58.097786 14156 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781935603120653-11672-0/minicluster-data/ts-0/data,/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781935603120653-11672-0/minicluster-data/ts-0/wal
uuid: "5ccd0d2539544400894bf073ec9a8538"
format_stamp: "Formatted at 2026-06-20 06:06:58 on dist-test-slave-9hm5"
server_key: "4d088aeb1b3791112e609dcc7d841538"
server_key_iv: "bdf43970c44b0d892ee3d4f265dd8520"
server_key_version: "encryptionkey@0"
I20260620 06:06:58.097892 14156 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781935603120653-11672-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781935603120653-11672-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781935603120653-11672-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:06:58.112843 14156 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:06:58.115975 14156 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:06:58.116175 14156 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:06:58.116737 14156 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 06:06:58.117516 14156 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 06:06:58.117587 14156 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 06:06:58.117655 14156 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 06:06:58.117690 14156 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 06:06:58.126488 14156 rpc_server.cc:307] RPC server started. Bound to: 127.11.102.1:43605
I20260620 06:06:58.126554 14284 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.102.1:43605 every 8 connection(s)
I20260620 06:06:58.127394 14156 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781935603120653-11672-0/minicluster-data/ts-0/data/info.pb
I20260620 06:06:58.130625 11672 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9HMp43/build/debug/bin/kudu as pid 14156
I20260620 06:06:58.130716 11672 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781935603120653-11672-0/minicluster-data/ts-0/wal/instance
I20260620 06:06:58.130995 11672 external_mini_cluster.cc:1428] Setting key 6722a0c1311dbb3b044ab7e657ae3f12
Jun 20 06:06:58 dist-test-slave-9hm5 krb5kdc[14049](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935618, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.102.1@KRBTEST.COM for kudu/127.11.102.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.11.102.2@KRBTEST.COM; defaulting to no policy
I20260620 06:06:58.140964 14133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:06:58.129150 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.102.1:47171 (local address 127.11.102.62:35025)
0620 06:06:58.129290 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0620 06:06:58.129295 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 06:06:58.130351 (+ 1056us) server_negotiation.cc:407] Connection header received
0620 06:06:58.131334 (+ 983us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:06:58.131338 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:06:58.131397 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:06:58.131474 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:06:58.132772 (+ 1298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:06:58.133289 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:06:58.134425 (+ 1136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:06:58.134666 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:06:58.134784 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:06:58.137336 (+ 2552us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:06:58.137345 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:06:58.137347 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:06:58.137370 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:06:58.138848 (+ 1478us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:06:58.139428 (+ 580us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:06:58.139435 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:06:58.139438 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:06:58.139522 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:06:58.139896 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:06:58.139901 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:06:58.139903 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:06:58.140096 (+ 193us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:06:58.140193 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0620 06:06:58.140673 (+ 480us) server_negotiation.cc:299] Negotiation successful
0620 06:06:58.140792 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260620 06:06:58.141815 14287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:06:58.129573 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.102.62:35025 (local address 127.11.102.1:47171)
0620 06:06:58.130188 (+ 615us) negotiation.cc:107] Waiting for socket to connect
0620 06:06:58.130226 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0620 06:06:58.131121 (+ 895us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:06:58.131682 (+ 561us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:06:58.131691 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:06:58.132138 (+ 447us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:06:58.132624 (+ 486us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:06:58.132637 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:06:58.133414 (+ 777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:06:58.133417 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:06:58.134298 (+ 881us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:06:58.134307 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:06:58.134822 (+ 515us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:06:58.134825 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:06:58.134999 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:06:58.135594 (+ 595us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:06:58.135665 (+ 71us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:06:58.137166 (+ 1501us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:06:58.138962 (+ 1796us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:06:58.138968 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:06:58.138983 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:06:58.139328 (+ 345us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:06:58.139642 (+ 314us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:06:58.139645 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:06:58.139647 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:06:58.139798 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:06:58.140191 (+ 393us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:06:58.140198 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:06:58.140439 (+ 241us) client_negotiation.cc:769] Sending connection context
0620 06:06:58.140618 (+ 179us) client_negotiation.cc:240] Negotiation successful
0620 06:06:58.140932 (+ 314us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":440,"thread_start_us":111,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.102.2@KRBTEST.COM" created.
I20260620 06:06:58.143215 14285 heartbeater.cc:347] Connected to a master server at 127.11.102.62:35025
I20260620 06:06:58.143452 14285 heartbeater.cc:464] Registering TS with master...
I20260620 06:06:58.143985 14285 heartbeater.cc:511] Master 127.11.102.62:35025 requested a full tablet report, sending...
I20260620 06:06:58.145426 14094 ts_manager.cc:194] Registered new tserver with Master: 5ccd0d2539544400894bf073ec9a8538 (127.11.102.1:43605)
I20260620 06:06:58.146502 14094 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.102.1@KRBTEST.COM'} at 127.11.102.1:47171
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.102.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781935603120653-11672-0/krb5kdc/kudu_127.11.102.2.keytab.
Entry for principal kudu/127.11.102.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781935603120653-11672-0/krb5kdc/kudu_127.11.102.2.keytab.
I20260620 06:06:58.158911 11672 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9HMp43/build/debug/bin/kudu
/tmp/dist-test-task9HMp43/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781935603120653-11672-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781935603120653-11672-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781935603120653-11672-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781935603120653-11672-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.102.2:0
--local_ip_for_outbound_sockets=127.11.102.2
--webserver_interface=127.11.102.2
--webserver_port=0
--tserver_master_addrs=127.11.102.62:35025
--keytab_file=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781935603120653-11672-0/krb5kdc/kudu_127.11.102.2.keytab
--principal=kudu/127.11.102.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.102.20:36841
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781935603120653-11672-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781935603120653-11672-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781935603120653-11672-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 06:06:58.266170 14290 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:06:58.266407 14290 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:06:58.266448 14290 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:06:58.270356 14290 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:06:58.270416 14290 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:06:58.270471 14290 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.102.2
I20260620 06:06:58.275413 14290 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.102.20:36841
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781935603120653-11672-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781935603120653-11672-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781935603120653-11672-0/krb5kdc/kudu_127.11.102.2.keytab
--principal=kudu/127.11.102.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.102.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781935603120653-11672-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.11.102.2
--webserver_port=0
--tserver_master_addrs=127.11.102.62:35025
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14290
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.102.2
--log_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781935603120653-11672-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 05:28:48 UTC on ccycloud.quasar-fixhms.root.comops.site
build id 4457020
I20260620 06:06:58.276391 14290 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:06:58.277294 14290 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 06:06:58.286545 14298 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:06:58.286553 14295 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:06:58.286553 14296 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:06:58.286733 14290 server_base.cc:1034] running on GCE node
I20260620 06:06:58.287168 14290 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:06:58.287871 14290 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:06:58.289052 14290 hybrid_clock.cc:648] HybridClock initialized: now 1781935618289025 us; error 54 us; skew 500 ppm
Jun 20 06:06:58 dist-test-slave-9hm5 krb5kdc[14049](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935618, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.102.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 06:06:58.291718 14290 init.cc:377] Logged in from keytab as kudu/127.11.102.2@KRBTEST.COM (short username kudu)
I20260620 06:06:58.292532 14290 webserver.cc:460] Webserver started at http://127.11.102.2:36257/ using document root <none> and password file <none>
I20260620 06:06:58.293088 14290 fs_manager.cc:359] Metadata directory not provided
I20260620 06:06:58.293176 14290 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:06:58.293406 14290 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 06:06:58.295547 14290 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781935603120653-11672-0/minicluster-data/ts-1/data/instance:
uuid: "4da7be92cfec4eacbacfa74eb1997327"
format_stamp: "Formatted at 2026-06-20 06:06:58 on dist-test-slave-9hm5"
server_key: "44ab0d9eb6d3e9f83a126d74854a1352"
server_key_iv: "bd48c1489841bd89c9ea003efcb2cd4f"
server_key_version: "encryptionkey@0"
I20260620 06:06:58.296176 14290 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781935603120653-11672-0/minicluster-data/ts-1/wal/instance:
uuid: "4da7be92cfec4eacbacfa74eb1997327"
format_stamp: "Formatted at 2026-06-20 06:06:58 on dist-test-slave-9hm5"
server_key: "44ab0d9eb6d3e9f83a126d74854a1352"
server_key_iv: "bd48c1489841bd89c9ea003efcb2cd4f"
server_key_version: "encryptionkey@0"
I20260620 06:06:58.299782 14290 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 06:06:58.302253 14305 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:06:58.303469 14290 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 06:06:58.303663 14290 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781935603120653-11672-0/minicluster-data/ts-1/data,/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781935603120653-11672-0/minicluster-data/ts-1/wal
uuid: "4da7be92cfec4eacbacfa74eb1997327"
format_stamp: "Formatted at 2026-06-20 06:06:58 on dist-test-slave-9hm5"
server_key: "44ab0d9eb6d3e9f83a126d74854a1352"
server_key_iv: "bd48c1489841bd89c9ea003efcb2cd4f"
server_key_version: "encryptionkey@0"
I20260620 06:06:58.303789 14290 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781935603120653-11672-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781935603120653-11672-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781935603120653-11672-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:06:58.313108 14290 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:06:58.315909 14290 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:06:58.316104 14290 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:06:58.316649 14290 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 06:06:58.317411 14290 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 06:06:58.317483 14290 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 06:06:58.317548 14290 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 06:06:58.317598 14290 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 06:06:58.326385 14290 rpc_server.cc:307] RPC server started. Bound to: 127.11.102.2:33807
I20260620 06:06:58.326416 14418 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.102.2:33807 every 8 connection(s)
I20260620 06:06:58.327236 14290 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781935603120653-11672-0/minicluster-data/ts-1/data/info.pb
I20260620 06:06:58.334880 11672 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9HMp43/build/debug/bin/kudu as pid 14290
I20260620 06:06:58.334992 11672 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781935603120653-11672-0/minicluster-data/ts-1/wal/instance
I20260620 06:06:58.335310 11672 external_mini_cluster.cc:1428] Setting key 6e8127b49cf9c3d21038475eaf603978
Jun 20 06:06:58 dist-test-slave-9hm5 krb5kdc[14049](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935618, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.102.2@KRBTEST.COM for kudu/127.11.102.62@KRBTEST.COM
I20260620 06:06:58.340781 14133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:06:58.329018 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.102.2:51089 (local address 127.11.102.62:35025)
0620 06:06:58.329178 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0620 06:06:58.329183 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 06:06:58.330265 (+ 1082us) server_negotiation.cc:407] Connection header received
0620 06:06:58.331075 (+ 810us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:06:58.331079 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:06:58.331134 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:06:58.331220 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:06:58.332369 (+ 1149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:06:58.332889 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:06:58.333965 (+ 1076us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:06:58.334174 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:06:58.334254 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:06:58.336927 (+ 2673us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:06:58.336931 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:06:58.336935 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:06:58.336962 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:06:58.338511 (+ 1549us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:06:58.339008 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:06:58.339013 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:06:58.339017 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:06:58.339110 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:06:58.339549 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:06:58.339555 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:06:58.339560 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:06:58.339802 (+ 242us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:06:58.339886 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0620 06:06:58.340378 (+ 492us) server_negotiation.cc:299] Negotiation successful
0620 06:06:58.340522 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260620 06:06:58.341435 14421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:06:58.329443 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.102.62:35025 (local address 127.11.102.2:51089)
0620 06:06:58.330081 (+ 638us) negotiation.cc:107] Waiting for socket to connect
0620 06:06:58.330120 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0620 06:06:58.330870 (+ 750us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:06:58.331368 (+ 498us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:06:58.331375 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:06:58.331726 (+ 351us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:06:58.332203 (+ 477us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:06:58.332217 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:06:58.332993 (+ 776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:06:58.332996 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:06:58.333854 (+ 858us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:06:58.333862 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:06:58.334296 (+ 434us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:06:58.334299 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:06:58.334430 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:06:58.335010 (+ 580us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:06:58.335034 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:06:58.336771 (+ 1737us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:06:58.338632 (+ 1861us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:06:58.338636 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:06:58.338648 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:06:58.338907 (+ 259us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:06:58.339205 (+ 298us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:06:58.339207 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:06:58.339209 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:06:58.339352 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:06:58.339896 (+ 544us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:06:58.339901 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:06:58.340123 (+ 222us) client_negotiation.cc:769] Sending connection context
0620 06:06:58.340302 (+ 179us) client_negotiation.cc:240] Negotiation successful
0620 06:06:58.340547 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":457,"thread_start_us":122,"threads_started":1}
I20260620 06:06:58.342778 14419 heartbeater.cc:347] Connected to a master server at 127.11.102.62:35025
I20260620 06:06:58.343020 14419 heartbeater.cc:464] Registering TS with master...
I20260620 06:06:58.343478 14419 heartbeater.cc:511] Master 127.11.102.62:35025 requested a full tablet report, sending...
I20260620 06:06:58.344522 14094 ts_manager.cc:194] Registered new tserver with Master: 4da7be92cfec4eacbacfa74eb1997327 (127.11.102.2:33807)
I20260620 06:06:58.345077 14094 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.102.2@KRBTEST.COM'} at 127.11.102.2:51089
WARNING: no policy specified for kudu/127.11.102.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.102.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.102.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781935603120653-11672-0/krb5kdc/kudu_127.11.102.3.keytab.
Entry for principal kudu/127.11.102.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781935603120653-11672-0/krb5kdc/kudu_127.11.102.3.keytab.
I20260620 06:06:58.362433 11672 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9HMp43/build/debug/bin/kudu
/tmp/dist-test-task9HMp43/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781935603120653-11672-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781935603120653-11672-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781935603120653-11672-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781935603120653-11672-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.102.3:0
--local_ip_for_outbound_sockets=127.11.102.3
--webserver_interface=127.11.102.3
--webserver_port=0
--tserver_master_addrs=127.11.102.62:35025
--keytab_file=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781935603120653-11672-0/krb5kdc/kudu_127.11.102.3.keytab
--principal=kudu/127.11.102.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.102.20:36841
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781935603120653-11672-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781935603120653-11672-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781935603120653-11672-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 06:06:58.469424 14424 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:06:58.469871 14424 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:06:58.469959 14424 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:06:58.477094 14424 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:06:58.477185 14424 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:06:58.477288 14424 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.102.3
I20260620 06:06:58.482872 14424 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.102.20:36841
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781935603120653-11672-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781935603120653-11672-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781935603120653-11672-0/krb5kdc/kudu_127.11.102.3.keytab
--principal=kudu/127.11.102.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.102.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781935603120653-11672-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.11.102.3
--webserver_port=0
--tserver_master_addrs=127.11.102.62:35025
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14424
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.102.3
--log_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781935603120653-11672-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 05:28:48 UTC on ccycloud.quasar-fixhms.root.comops.site
build id 4457020
I20260620 06:06:58.483950 14424 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:06:58.484874 14424 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 06:06:58.493968 14430 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:06:58.494014 14429 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:06:58.494159 14424 server_base.cc:1034] running on GCE node
W20260620 06:06:58.494024 14432 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:06:58.494596 14424 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:06:58.495230 14424 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:06:58.496383 14424 hybrid_clock.cc:648] HybridClock initialized: now 1781935618496332 us; error 65 us; skew 500 ppm
Jun 20 06:06:58 dist-test-slave-9hm5 krb5kdc[14049](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935618, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.102.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 06:06:58.499178 14424 init.cc:377] Logged in from keytab as kudu/127.11.102.3@KRBTEST.COM (short username kudu)
I20260620 06:06:58.500051 14424 webserver.cc:460] Webserver started at http://127.11.102.3:36813/ using document root <none> and password file <none>
I20260620 06:06:58.500605 14424 fs_manager.cc:359] Metadata directory not provided
I20260620 06:06:58.500665 14424 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:06:58.500903 14424 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 06:06:58.502949 14424 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781935603120653-11672-0/minicluster-data/ts-2/data/instance:
uuid: "6f9afcd0ef9b4432be087a36eced2fd8"
format_stamp: "Formatted at 2026-06-20 06:06:58 on dist-test-slave-9hm5"
server_key: "b7d9cb9d09a951b68a518c61089d5ace"
server_key_iv: "eebbd79f52c4bf275bf3209b44c808d8"
server_key_version: "encryptionkey@0"
I20260620 06:06:58.503496 14424 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781935603120653-11672-0/minicluster-data/ts-2/wal/instance:
uuid: "6f9afcd0ef9b4432be087a36eced2fd8"
format_stamp: "Formatted at 2026-06-20 06:06:58 on dist-test-slave-9hm5"
server_key: "b7d9cb9d09a951b68a518c61089d5ace"
server_key_iv: "eebbd79f52c4bf275bf3209b44c808d8"
server_key_version: "encryptionkey@0"
I20260620 06:06:58.507139 14424 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260620 06:06:58.509613 14439 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:06:58.510800 14424 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 06:06:58.511029 14424 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781935603120653-11672-0/minicluster-data/ts-2/data,/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781935603120653-11672-0/minicluster-data/ts-2/wal
uuid: "6f9afcd0ef9b4432be087a36eced2fd8"
format_stamp: "Formatted at 2026-06-20 06:06:58 on dist-test-slave-9hm5"
server_key: "b7d9cb9d09a951b68a518c61089d5ace"
server_key_iv: "eebbd79f52c4bf275bf3209b44c808d8"
server_key_version: "encryptionkey@0"
I20260620 06:06:58.511147 14424 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781935603120653-11672-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781935603120653-11672-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781935603120653-11672-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:06:58.526930 14424 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:06:58.529685 14424 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:06:58.529879 14424 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:06:58.530437 14424 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 06:06:58.531211 14424 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 06:06:58.531256 14424 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 06:06:58.531316 14424 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 06:06:58.531349 14424 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 06:06:58.539458 14424 rpc_server.cc:307] RPC server started. Bound to: 127.11.102.3:39327
I20260620 06:06:58.539487 14552 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.102.3:39327 every 8 connection(s)
I20260620 06:06:58.540398 14424 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781935603120653-11672-0/minicluster-data/ts-2/data/info.pb
I20260620 06:06:58.548015 11672 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9HMp43/build/debug/bin/kudu as pid 14424
I20260620 06:06:58.548153 11672 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781935603120653-11672-0/minicluster-data/ts-2/wal/instance
I20260620 06:06:58.548429 11672 external_mini_cluster.cc:1428] Setting key 9df3e1b723837b9ca07ba64b22b770e4
Jun 20 06:06:58 dist-test-slave-9hm5 krb5kdc[14049](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935618, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.102.3@KRBTEST.COM for kudu/127.11.102.62@KRBTEST.COM
I20260620 06:06:58.553471 14133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:06:58.542089 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.102.3:48837 (local address 127.11.102.62:35025)
0620 06:06:58.542237 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0620 06:06:58.542241 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 06:06:58.543451 (+ 1210us) server_negotiation.cc:407] Connection header received
0620 06:06:58.544335 (+ 884us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:06:58.544338 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:06:58.544384 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:06:58.544457 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:06:58.545532 (+ 1075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:06:58.546051 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:06:58.547079 (+ 1028us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:06:58.547288 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:06:58.547362 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:06:58.549930 (+ 2568us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:06:58.549934 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:06:58.549936 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:06:58.549962 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:06:58.551462 (+ 1500us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:06:58.552008 (+ 546us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:06:58.552013 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:06:58.552014 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:06:58.552123 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:06:58.552456 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:06:58.552459 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:06:58.552461 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:06:58.552637 (+ 176us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:06:58.552740 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0620 06:06:58.553199 (+ 459us) server_negotiation.cc:299] Negotiation successful
0620 06:06:58.553310 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260620 06:06:58.554175 14555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:06:58.542518 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.102.62:35025 (local address 127.11.102.3:48837)
0620 06:06:58.543250 (+ 732us) negotiation.cc:107] Waiting for socket to connect
0620 06:06:58.543290 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0620 06:06:58.544159 (+ 869us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:06:58.544625 (+ 466us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:06:58.544631 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:06:58.544960 (+ 329us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:06:58.545408 (+ 448us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:06:58.545421 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:06:58.546149 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:06:58.546151 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:06:58.546984 (+ 833us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:06:58.546991 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:06:58.547405 (+ 414us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:06:58.547408 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:06:58.547538 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:06:58.548271 (+ 733us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:06:58.548286 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:06:58.549790 (+ 1504us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:06:58.551639 (+ 1849us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:06:58.551644 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:06:58.551658 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:06:58.551903 (+ 245us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:06:58.552217 (+ 314us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:06:58.552222 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:06:58.552224 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:06:58.552353 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:06:58.552738 (+ 385us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:06:58.552745 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:06:58.552967 (+ 222us) client_negotiation.cc:769] Sending connection context
0620 06:06:58.553143 (+ 176us) client_negotiation.cc:240] Negotiation successful
0620 06:06:58.553430 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":535,"thread_start_us":111,"threads_started":1}
I20260620 06:06:58.555176 14553 heartbeater.cc:347] Connected to a master server at 127.11.102.62:35025
I20260620 06:06:58.555415 14553 heartbeater.cc:464] Registering TS with master...
I20260620 06:06:58.555873 14553 heartbeater.cc:511] Master 127.11.102.62:35025 requested a full tablet report, sending...
I20260620 06:06:58.556807 14094 ts_manager.cc:194] Registered new tserver with Master: 6f9afcd0ef9b4432be087a36eced2fd8 (127.11.102.3:39327)
I20260620 06:06:58.557348 14094 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.102.3@KRBTEST.COM'} at 127.11.102.3:48837
I20260620 06:06:58.561998 11672 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260620 06:06:58.574124 14133 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 06:06:58.572870 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44090 (local address 127.11.102.62:35025)
0620 06:06:58.572993 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0620 06:06:58.572997 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 06:06:58.573175 (+ 178us) server_negotiation.cc:407] Connection header received
0620 06:06:58.573271 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:06:58.573274 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:06:58.573314 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:06:58.573386 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:06:58.573979 (+ 593us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44090: BlockingRecv error: recv got EOF from 127.0.0.1:44090 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
I20260620 06:06:58.575186 11672 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9HMp43/build/debug/bin/kudu with pid 14156
I20260620 06:06:58.580554 11672 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9HMp43/build/debug/bin/kudu with pid 14290
I20260620 06:06:58.586099 11672 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9HMp43/build/debug/bin/kudu with pid 14424
I20260620 06:06:58.591172 11672 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9HMp43/build/debug/bin/kudu with pid 14062
2026-06-20T06:06:58Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (2053 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781935603120653-11672-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 06:06:58 dist-test-slave-9hm5 krb5kdc[14565](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 06:06:58 dist-test-slave-9hm5 krb5kdc[14565](info): set up 2 sockets
Jun 20 06:06:58 dist-test-slave-9hm5 krb5kdc[14565](info): commencing operation
krb5kdc: starting...
W20260620 06:06:59.643903 11672 mini_kdc.cc:121] Time spent starting KDC: real 1.031s user 0.005s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781935603120653-11672-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781935603120653-11672-0/krb5kdc/test-user.keytab.
Jun 20 06:06:59 dist-test-slave-9hm5 krb5kdc[14565](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935619, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T06:06:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T06:06:59Z Disabled control of system clock
WARNING: no policy specified for kudu/127.11.102.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.102.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.102.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781935603120653-11672-0/krb5kdc/kudu_127.11.102.62.keytab.
Entry for principal kudu/127.11.102.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781935603120653-11672-0/krb5kdc/kudu_127.11.102.62.keytab.
I20260620 06:06:59.762398 11672 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9HMp43/build/debug/bin/kudu
/tmp/dist-test-task9HMp43/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781935603120653-11672-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781935603120653-11672-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781935603120653-11672-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781935603120653-11672-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.102.62:40873
--webserver_interface=127.11.102.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781935603120653-11672-0/krb5kdc/kudu_127.11.102.62.keytab
--principal=kudu/127.11.102.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.102.20:44819
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.102.62:40873
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781935603120653-11672-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781935603120653-11672-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781935603120653-11672-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 06:06:59.868683 14578 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:06:59.868975 14578 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:06:59.869067 14578 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:06:59.873158 14578 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 06:06:59.873245 14578 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:06:59.873286 14578 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:06:59.873342 14578 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 06:06:59.873383 14578 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 06:06:59.878518 14578 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.102.20:44819
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781935603120653-11672-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781935603120653-11672-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.102.62:40873
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781935603120653-11672-0/krb5kdc/kudu_127.11.102.62.keytab
--principal=kudu/127.11.102.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.102.62:40873
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781935603120653-11672-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.102.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14578
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781935603120653-11672-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 05:28:48 UTC on ccycloud.quasar-fixhms.root.comops.site
build id 4457020
I20260620 06:06:59.879565 14578 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:06:59.880599 14578 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 06:06:59.888612 14583 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:06:59.888677 14584 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:06:59.888639 14586 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:06:59.889024 14578 server_base.cc:1034] running on GCE node
I20260620 06:06:59.889556 14578 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:06:59.890597 14578 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:06:59.891770 14578 hybrid_clock.cc:648] HybridClock initialized: now 1781935619891757 us; error 24 us; skew 500 ppm
Jun 20 06:06:59 dist-test-slave-9hm5 krb5kdc[14565](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935619, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.102.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 06:06:59.894428 14578 init.cc:377] Logged in from keytab as kudu/127.11.102.62@KRBTEST.COM (short username kudu)
I20260620 06:06:59.895221 14578 webserver.cc:460] Webserver started at http://127.11.102.62:34991/ using document root <none> and password file <none>
I20260620 06:06:59.895778 14578 fs_manager.cc:359] Metadata directory not provided
I20260620 06:06:59.895844 14578 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:06:59.896015 14578 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 06:06:59.897954 14578 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781935603120653-11672-0/minicluster-data/master-0/data/instance:
uuid: "839fe1e826524a38b2cceb9218f99485"
format_stamp: "Formatted at 2026-06-20 06:06:59 on dist-test-slave-9hm5"
server_key: "8b244da397192a728d12eb2a714015e3"
server_key_iv: "d88873f5626f4ee0c124c9298c1924c6"
server_key_version: "encryptionkey@0"
I20260620 06:06:59.898450 14578 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781935603120653-11672-0/minicluster-data/master-0/wal/instance:
uuid: "839fe1e826524a38b2cceb9218f99485"
format_stamp: "Formatted at 2026-06-20 06:06:59 on dist-test-slave-9hm5"
server_key: "8b244da397192a728d12eb2a714015e3"
server_key_iv: "d88873f5626f4ee0c124c9298c1924c6"
server_key_version: "encryptionkey@0"
I20260620 06:06:59.902014 14578 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260620 06:06:59.904400 14593 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:06:59.905567 14578 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 06:06:59.905694 14578 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781935603120653-11672-0/minicluster-data/master-0/data,/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781935603120653-11672-0/minicluster-data/master-0/wal
uuid: "839fe1e826524a38b2cceb9218f99485"
format_stamp: "Formatted at 2026-06-20 06:06:59 on dist-test-slave-9hm5"
server_key: "8b244da397192a728d12eb2a714015e3"
server_key_iv: "d88873f5626f4ee0c124c9298c1924c6"
server_key_version: "encryptionkey@0"
I20260620 06:06:59.905795 14578 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781935603120653-11672-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781935603120653-11672-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781935603120653-11672-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:06:59.923137 14578 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:06:59.926304 14578 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:06:59.926518 14578 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:06:59.933645 14578 rpc_server.cc:307] RPC server started. Bound to: 127.11.102.62:40873
I20260620 06:06:59.933913 14645 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.102.62:40873 every 8 connection(s)
I20260620 06:06:59.934646 14578 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781935603120653-11672-0/minicluster-data/master-0/data/info.pb
I20260620 06:06:59.937489 14646 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 06:06:59.938110 11672 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9HMp43/build/debug/bin/kudu as pid 14578
I20260620 06:06:59.938222 11672 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781935603120653-11672-0/minicluster-data/master-0/wal/instance
I20260620 06:06:59.938519 11672 external_mini_cluster.cc:1428] Setting key a10e6789bd330058a738c1005b6a3fc9
I20260620 06:06:59.943972 14646 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 839fe1e826524a38b2cceb9218f99485: Bootstrap starting.
I20260620 06:06:59.946604 14646 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 839fe1e826524a38b2cceb9218f99485: Neither blocks nor log segments found. Creating new log.
I20260620 06:06:59.947477 14646 log.cc:824] T 00000000000000000000000000000000 P 839fe1e826524a38b2cceb9218f99485: Log is configured to *not* fsync() on all Append() calls
Jun 20 06:06:59 dist-test-slave-9hm5 krb5kdc[14565](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935619, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.102.62@KRBTEST.COM
I20260620 06:06:59.949343 14646 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 839fe1e826524a38b2cceb9218f99485: No bootstrap required, opened a new log
I20260620 06:06:59.952986 14649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:06:59.939901 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59670 (local address 127.11.102.62:40873)
0620 06:06:59.940297 (+ 396us) server_negotiation.cc:206] Beginning negotiation
0620 06:06:59.940312 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 06:06:59.940347 (+ 35us) server_negotiation.cc:407] Connection header received
0620 06:06:59.940986 (+ 639us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:06:59.941003 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:06:59.941326 (+ 323us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:06:59.941612 (+ 286us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:06:59.942513 (+ 901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:06:59.943760 (+ 1247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:06:59.944871 (+ 1111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:06:59.945341 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:06:59.945468 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:06:59.948329 (+ 2861us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:06:59.948334 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:06:59.948347 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:06:59.948374 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:06:59.950147 (+ 1773us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:06:59.950613 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:06:59.950619 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:06:59.950623 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:06:59.950730 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:06:59.951026 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:06:59.951031 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:06:59.951032 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:06:59.951451 (+ 419us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:06:59.951667 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0620 06:06:59.951968 (+ 301us) server_negotiation.cc:299] Negotiation successful
0620 06:06:59.952249 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":246,"thread_start_us":124,"threads_started":1}
I20260620 06:06:59.952852 14646 raft_consensus.cc:348] T 00000000000000000000000000000000 P 839fe1e826524a38b2cceb9218f99485 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "839fe1e826524a38b2cceb9218f99485" member_type: VOTER last_known_addr { host: "127.11.102.62" port: 40873 } }
I20260620 06:06:59.953109 14646 raft_consensus.cc:374] T 00000000000000000000000000000000 P 839fe1e826524a38b2cceb9218f99485 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 06:06:59.953171 14646 raft_consensus.cc:729] T 00000000000000000000000000000000 P 839fe1e826524a38b2cceb9218f99485 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 839fe1e826524a38b2cceb9218f99485, State: Initialized, Role: FOLLOWER
I20260620 06:06:59.953603 14646 consensus_queue.cc:260] T 00000000000000000000000000000000 P 839fe1e826524a38b2cceb9218f99485 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "839fe1e826524a38b2cceb9218f99485" member_type: VOTER last_known_addr { host: "127.11.102.62" port: 40873 } }
I20260620 06:06:59.953751 14646 raft_consensus.cc:388] T 00000000000000000000000000000000 P 839fe1e826524a38b2cceb9218f99485 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 06:06:59.953820 14646 raft_consensus.cc:482] T 00000000000000000000000000000000 P 839fe1e826524a38b2cceb9218f99485 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 06:06:59.953917 14646 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 839fe1e826524a38b2cceb9218f99485 [term 0 FOLLOWER]: Advancing to term 1
I20260620 06:06:59.954838 14646 raft_consensus.cc:504] T 00000000000000000000000000000000 P 839fe1e826524a38b2cceb9218f99485 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "839fe1e826524a38b2cceb9218f99485" member_type: VOTER last_known_addr { host: "127.11.102.62" port: 40873 } }
I20260620 06:06:59.955202 14646 leader_election.cc:302] T 00000000000000000000000000000000 P 839fe1e826524a38b2cceb9218f99485 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 839fe1e826524a38b2cceb9218f99485; no voters:
I20260620 06:06:59.955559 14646 leader_election.cc:288] T 00000000000000000000000000000000 P 839fe1e826524a38b2cceb9218f99485 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 06:06:59.955664 14651 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 839fe1e826524a38b2cceb9218f99485 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 06:06:59.955906 14651 raft_consensus.cc:686] T 00000000000000000000000000000000 P 839fe1e826524a38b2cceb9218f99485 [term 1 LEADER]: Becoming Leader. State: Replica: 839fe1e826524a38b2cceb9218f99485, State: Running, Role: LEADER
I20260620 06:06:59.955952 11672 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 06:06:59.956202 14651 consensus_queue.cc:237] T 00000000000000000000000000000000 P 839fe1e826524a38b2cceb9218f99485 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "839fe1e826524a38b2cceb9218f99485" member_type: VOTER last_known_addr { host: "127.11.102.62" port: 40873 } }
I20260620 06:06:59.956689 14646 sys_catalog.cc:564] T 00000000000000000000000000000000 P 839fe1e826524a38b2cceb9218f99485 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 06:06:59.957398 14651 sys_catalog.cc:455] T 00000000000000000000000000000000 P 839fe1e826524a38b2cceb9218f99485 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 839fe1e826524a38b2cceb9218f99485. Latest consensus state: current_term: 1 leader_uuid: "839fe1e826524a38b2cceb9218f99485" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "839fe1e826524a38b2cceb9218f99485" member_type: VOTER last_known_addr { host: "127.11.102.62" port: 40873 } } }
I20260620 06:06:59.957512 14651 sys_catalog.cc:458] T 00000000000000000000000000000000 P 839fe1e826524a38b2cceb9218f99485 [sys.catalog]: This master's current role is: LEADER
I20260620 06:06:59.958164 14653 sys_catalog.cc:455] T 00000000000000000000000000000000 P 839fe1e826524a38b2cceb9218f99485 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "839fe1e826524a38b2cceb9218f99485" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "839fe1e826524a38b2cceb9218f99485" member_type: VOTER last_known_addr { host: "127.11.102.62" port: 40873 } } }
I20260620 06:06:59.958250 14653 sys_catalog.cc:458] T 00000000000000000000000000000000 P 839fe1e826524a38b2cceb9218f99485 [sys.catalog]: This master's current role is: LEADER
I20260620 06:06:59.958531 14660 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 06:06:59.961520 14660 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 06:06:59.967128 14660 catalog_manager.cc:1347] Generated new cluster ID: 9858507bb7c24770b2fef14c7bbacd21
I20260620 06:06:59.967186 14660 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 06:06:59.978122 14649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:06:59.972663 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59676 (local address 127.11.102.62:40873)
0620 06:06:59.972765 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0620 06:06:59.972769 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 06:06:59.973033 (+ 264us) server_negotiation.cc:407] Connection header received
0620 06:06:59.973134 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:06:59.973138 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:06:59.973191 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:06:59.973281 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:06:59.973810 (+ 529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:06:59.974654 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:06:59.975777 (+ 1123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:06:59.976137 (+ 360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:06:59.976243 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:06:59.977121 (+ 878us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:06:59.977125 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:06:59.977128 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 06:06:59.977680 (+ 552us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:06:59.977714 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 06:06:59.977733 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:06:59.977821 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0620 06:06:59.977921 (+ 100us) server_negotiation.cc:299] Negotiation successful
0620 06:06:59.977978 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260620 06:06:59.981223 14660 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 06:06:59.982085 14660 catalog_manager.cc:1504] Loading token signing keys...
I20260620 06:06:59.993319 14660 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 839fe1e826524a38b2cceb9218f99485: Generated new TSK 0
I20260620 06:06:59.994021 14660 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 06:07:00.000069 11672 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9HMp43/build/debug/bin/kudu with pid 14578
2026-06-20T06:07:00Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1406 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781935603120653-11672-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 06:07:00 dist-test-slave-9hm5 krb5kdc[14679](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 06:07:00 dist-test-slave-9hm5 krb5kdc[14679](info): set up 2 sockets
Jun 20 06:07:00 dist-test-slave-9hm5 krb5kdc[14679](info): commencing operation
krb5kdc: starting...
W20260620 06:07:01.031539 11672 mini_kdc.cc:121] Time spent starting KDC: real 1.012s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781935603120653-11672-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781935603120653-11672-0/krb5kdc/test-user.keytab.
Jun 20 06:07:01 dist-test-slave-9hm5 krb5kdc[14679](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935621, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T06:07:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T06:07:01Z Disabled control of system clock
WARNING: no policy specified for kudu/127.11.102.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.102.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.102.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781935603120653-11672-0/krb5kdc/kudu_127.11.102.62.keytab.
Entry for principal kudu/127.11.102.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781935603120653-11672-0/krb5kdc/kudu_127.11.102.62.keytab.
I20260620 06:07:01.148257 11672 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9HMp43/build/debug/bin/kudu
/tmp/dist-test-task9HMp43/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781935603120653-11672-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781935603120653-11672-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781935603120653-11672-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781935603120653-11672-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.102.62:41431
--webserver_interface=127.11.102.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781935603120653-11672-0/krb5kdc/kudu_127.11.102.62.keytab
--principal=kudu/127.11.102.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.102.20:45769
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.102.62:41431
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781935603120653-11672-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781935603120653-11672-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781935603120653-11672-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 06:07:01.252976 14692 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:07:01.253244 14692 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:07:01.253317 14692 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:07:01.257303 14692 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 06:07:01.257380 14692 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:07:01.257417 14692 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:07:01.257473 14692 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 06:07:01.257512 14692 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 06:07:01.262553 14692 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.102.20:45769
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781935603120653-11672-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781935603120653-11672-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.102.62:41431
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781935603120653-11672-0/krb5kdc/kudu_127.11.102.62.keytab
--principal=kudu/127.11.102.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.102.62:41431
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781935603120653-11672-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.102.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14692
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781935603120653-11672-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 05:28:48 UTC on ccycloud.quasar-fixhms.root.comops.site
build id 4457020
I20260620 06:07:01.263486 14692 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:07:01.264473 14692 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 06:07:01.272631 14698 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:07:01.272678 14697 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:07:01.272668 14700 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:07:01.272732 14692 server_base.cc:1034] running on GCE node
I20260620 06:07:01.273418 14692 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:07:01.274410 14692 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:07:01.275571 14692 hybrid_clock.cc:648] HybridClock initialized: now 1781935621275560 us; error 27 us; skew 500 ppm
Jun 20 06:07:01 dist-test-slave-9hm5 krb5kdc[14679](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935621, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.102.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 06:07:01.278173 14692 init.cc:377] Logged in from keytab as kudu/127.11.102.62@KRBTEST.COM (short username kudu)
I20260620 06:07:01.279004 14692 webserver.cc:460] Webserver started at http://127.11.102.62:42175/ using document root <none> and password file <none>
I20260620 06:07:01.279520 14692 fs_manager.cc:359] Metadata directory not provided
I20260620 06:07:01.279605 14692 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:07:01.279857 14692 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 06:07:01.281730 14692 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781935603120653-11672-0/minicluster-data/master-0/data/instance:
uuid: "d4aac321fced4e6483e607c5e0ef534a"
format_stamp: "Formatted at 2026-06-20 06:07:01 on dist-test-slave-9hm5"
server_key: "d4e57feb53b7ed039758c81bec34f910"
server_key_iv: "cdb335b9b5ec94f24527657edef4e0f2"
server_key_version: "encryptionkey@0"
I20260620 06:07:01.282262 14692 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781935603120653-11672-0/minicluster-data/master-0/wal/instance:
uuid: "d4aac321fced4e6483e607c5e0ef534a"
format_stamp: "Formatted at 2026-06-20 06:07:01 on dist-test-slave-9hm5"
server_key: "d4e57feb53b7ed039758c81bec34f910"
server_key_iv: "cdb335b9b5ec94f24527657edef4e0f2"
server_key_version: "encryptionkey@0"
I20260620 06:07:01.285677 14692 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.004s
I20260620 06:07:01.287938 14707 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:07:01.289116 14692 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.001s
I20260620 06:07:01.289256 14692 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781935603120653-11672-0/minicluster-data/master-0/data,/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781935603120653-11672-0/minicluster-data/master-0/wal
uuid: "d4aac321fced4e6483e607c5e0ef534a"
format_stamp: "Formatted at 2026-06-20 06:07:01 on dist-test-slave-9hm5"
server_key: "d4e57feb53b7ed039758c81bec34f910"
server_key_iv: "cdb335b9b5ec94f24527657edef4e0f2"
server_key_version: "encryptionkey@0"
I20260620 06:07:01.289359 14692 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781935603120653-11672-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781935603120653-11672-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781935603120653-11672-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:07:01.306154 14692 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:07:01.308996 14692 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:07:01.309239 14692 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:07:01.316121 14692 rpc_server.cc:307] RPC server started. Bound to: 127.11.102.62:41431
I20260620 06:07:01.316393 14759 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.102.62:41431 every 8 connection(s)
I20260620 06:07:01.317062 14692 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781935603120653-11672-0/minicluster-data/master-0/data/info.pb
I20260620 06:07:01.319721 14760 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 06:07:01.323302 11672 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9HMp43/build/debug/bin/kudu as pid 14692
I20260620 06:07:01.323424 11672 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781935603120653-11672-0/minicluster-data/master-0/wal/instance
I20260620 06:07:01.323726 11672 external_mini_cluster.cc:1428] Setting key fecf55c1799dc729bd72e231c61ed33a
I20260620 06:07:01.325354 14760 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d4aac321fced4e6483e607c5e0ef534a: Bootstrap starting.
I20260620 06:07:01.327754 14760 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d4aac321fced4e6483e607c5e0ef534a: Neither blocks nor log segments found. Creating new log.
I20260620 06:07:01.328459 14760 log.cc:824] T 00000000000000000000000000000000 P d4aac321fced4e6483e607c5e0ef534a: Log is configured to *not* fsync() on all Append() calls
I20260620 06:07:01.330477 14760 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d4aac321fced4e6483e607c5e0ef534a: No bootstrap required, opened a new log
Jun 20 06:07:01 dist-test-slave-9hm5 krb5kdc[14679](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935621, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.102.62@KRBTEST.COM
I20260620 06:07:01.334520 14760 raft_consensus.cc:348] T 00000000000000000000000000000000 P d4aac321fced4e6483e607c5e0ef534a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4aac321fced4e6483e607c5e0ef534a" member_type: VOTER last_known_addr { host: "127.11.102.62" port: 41431 } }
I20260620 06:07:01.334750 14760 raft_consensus.cc:374] T 00000000000000000000000000000000 P d4aac321fced4e6483e607c5e0ef534a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 06:07:01.334843 14760 raft_consensus.cc:729] T 00000000000000000000000000000000 P d4aac321fced4e6483e607c5e0ef534a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d4aac321fced4e6483e607c5e0ef534a, State: Initialized, Role: FOLLOWER
I20260620 06:07:01.335256 14760 consensus_queue.cc:260] T 00000000000000000000000000000000 P d4aac321fced4e6483e607c5e0ef534a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4aac321fced4e6483e607c5e0ef534a" member_type: VOTER last_known_addr { host: "127.11.102.62" port: 41431 } }
I20260620 06:07:01.335410 14760 raft_consensus.cc:388] T 00000000000000000000000000000000 P d4aac321fced4e6483e607c5e0ef534a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 06:07:01.335500 14760 raft_consensus.cc:482] T 00000000000000000000000000000000 P d4aac321fced4e6483e607c5e0ef534a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 06:07:01.335641 14760 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d4aac321fced4e6483e607c5e0ef534a [term 0 FOLLOWER]: Advancing to term 1
I20260620 06:07:01.336486 14760 raft_consensus.cc:504] T 00000000000000000000000000000000 P d4aac321fced4e6483e607c5e0ef534a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4aac321fced4e6483e607c5e0ef534a" member_type: VOTER last_known_addr { host: "127.11.102.62" port: 41431 } }
I20260620 06:07:01.336818 14760 leader_election.cc:302] T 00000000000000000000000000000000 P d4aac321fced4e6483e607c5e0ef534a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d4aac321fced4e6483e607c5e0ef534a; no voters:
I20260620 06:07:01.337138 14760 leader_election.cc:288] T 00000000000000000000000000000000 P d4aac321fced4e6483e607c5e0ef534a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 06:07:01.337199 14765 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d4aac321fced4e6483e607c5e0ef534a [term 1 FOLLOWER]: Leader election won for term 1
I20260620 06:07:01.337417 14765 raft_consensus.cc:686] T 00000000000000000000000000000000 P d4aac321fced4e6483e607c5e0ef534a [term 1 LEADER]: Becoming Leader. State: Replica: d4aac321fced4e6483e607c5e0ef534a, State: Running, Role: LEADER
I20260620 06:07:01.337750 14765 consensus_queue.cc:237] T 00000000000000000000000000000000 P d4aac321fced4e6483e607c5e0ef534a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4aac321fced4e6483e607c5e0ef534a" member_type: VOTER last_known_addr { host: "127.11.102.62" port: 41431 } }
I20260620 06:07:01.338398 14760 sys_catalog.cc:564] T 00000000000000000000000000000000 P d4aac321fced4e6483e607c5e0ef534a [sys.catalog]: configured and running, proceeding with master startup.
I20260620 06:07:01.338658 14763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:07:01.325081 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54338 (local address 127.11.102.62:41431)
0620 06:07:01.325500 (+ 419us) server_negotiation.cc:206] Beginning negotiation
0620 06:07:01.325519 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0620 06:07:01.325576 (+ 57us) server_negotiation.cc:407] Connection header received
0620 06:07:01.326197 (+ 621us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:07:01.326213 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:07:01.326526 (+ 313us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:07:01.326808 (+ 282us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:07:01.327704 (+ 896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:07:01.328868 (+ 1164us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:07:01.329998 (+ 1130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:07:01.330459 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:07:01.330641 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:07:01.333420 (+ 2779us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:07:01.333425 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:07:01.333437 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:07:01.333464 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:07:01.335491 (+ 2027us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:07:01.336033 (+ 542us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:07:01.336040 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:07:01.336045 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:07:01.336173 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:07:01.336498 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:07:01.336502 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:07:01.336504 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:07:01.336976 (+ 472us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:07:01.337221 (+ 245us) server_negotiation.cc:1035] Waiting for connection context
0620 06:07:01.337478 (+ 257us) server_negotiation.cc:299] Negotiation successful
0620 06:07:01.337808 (+ 330us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":276,"thread_start_us":151,"threads_started":1}
I20260620 06:07:01.338907 14765 sys_catalog.cc:455] T 00000000000000000000000000000000 P d4aac321fced4e6483e607c5e0ef534a [sys.catalog]: SysCatalogTable state changed. Reason: New leader d4aac321fced4e6483e607c5e0ef534a. Latest consensus state: current_term: 1 leader_uuid: "d4aac321fced4e6483e607c5e0ef534a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4aac321fced4e6483e607c5e0ef534a" member_type: VOTER last_known_addr { host: "127.11.102.62" port: 41431 } } }
I20260620 06:07:01.339092 14765 sys_catalog.cc:458] T 00000000000000000000000000000000 P d4aac321fced4e6483e607c5e0ef534a [sys.catalog]: This master's current role is: LEADER
I20260620 06:07:01.339337 14765 sys_catalog.cc:455] T 00000000000000000000000000000000 P d4aac321fced4e6483e607c5e0ef534a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d4aac321fced4e6483e607c5e0ef534a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4aac321fced4e6483e607c5e0ef534a" member_type: VOTER last_known_addr { host: "127.11.102.62" port: 41431 } } }
I20260620 06:07:01.339476 14765 sys_catalog.cc:458] T 00000000000000000000000000000000 P d4aac321fced4e6483e607c5e0ef534a [sys.catalog]: This master's current role is: LEADER
W20260620 06:07:01.343111 14780 catalog_manager.cc:1558] T 00000000000000000000000000000000 P d4aac321fced4e6483e607c5e0ef534a: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 06:07:01.343210 14780 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 06:07:01.343286 14772 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 06:07:01.344009 14772 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 06:07:01.348809 14772 catalog_manager.cc:1347] Generated new cluster ID: e0fa3098bad14bd9a60d0469115fa80d
I20260620 06:07:01.348870 14772 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 06:07:01.359026 14772 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 06:07:01.360067 14772 catalog_manager.cc:1504] Loading token signing keys...
I20260620 06:07:01.372905 14772 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d4aac321fced4e6483e607c5e0ef534a: Generated new TSK 0
I20260620 06:07:01.373543 14772 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.11.102.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.102.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.102.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781935603120653-11672-0/krb5kdc/kudu_127.11.102.1.keytab.
Entry for principal kudu/127.11.102.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781935603120653-11672-0/krb5kdc/kudu_127.11.102.1.keytab.
I20260620 06:07:01.404819 11672 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9HMp43/build/debug/bin/kudu
/tmp/dist-test-task9HMp43/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781935603120653-11672-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781935603120653-11672-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781935603120653-11672-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781935603120653-11672-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.102.1:0
--local_ip_for_outbound_sockets=127.11.102.1
--webserver_interface=127.11.102.1
--webserver_port=0
--tserver_master_addrs=127.11.102.62:41431
--keytab_file=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781935603120653-11672-0/krb5kdc/kudu_127.11.102.1.keytab
--principal=kudu/127.11.102.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.102.20:45769
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781935603120653-11672-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781935603120653-11672-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781935603120653-11672-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 06:07:01.521987 14786 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:07:01.522238 14786 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:07:01.522284 14786 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:07:01.526254 14786 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:07:01.526312 14786 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:07:01.526417 14786 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.102.1
I20260620 06:07:01.531587 14786 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.102.20:45769
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781935603120653-11672-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781935603120653-11672-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781935603120653-11672-0/krb5kdc/kudu_127.11.102.1.keytab
--principal=kudu/127.11.102.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.102.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781935603120653-11672-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.11.102.1
--webserver_port=0
--tserver_master_addrs=127.11.102.62:41431
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14786
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.102.1
--log_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781935603120653-11672-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 05:28:48 UTC on ccycloud.quasar-fixhms.root.comops.site
build id 4457020
I20260620 06:07:01.532598 14786 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:07:01.533600 14786 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 06:07:01.543269 14794 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:07:01.543368 14786 server_base.cc:1034] running on GCE node
W20260620 06:07:01.543282 14792 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:07:01.543294 14791 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:07:01.543967 14786 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:07:01.544607 14786 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:07:01.545799 14786 hybrid_clock.cc:648] HybridClock initialized: now 1781935621545788 us; error 34 us; skew 500 ppm
Jun 20 06:07:01 dist-test-slave-9hm5 krb5kdc[14679](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935621, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.102.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 06:07:01.548595 14786 init.cc:377] Logged in from keytab as kudu/127.11.102.1@KRBTEST.COM (short username kudu)
I20260620 06:07:01.549377 14786 webserver.cc:460] Webserver started at http://127.11.102.1:39383/ using document root <none> and password file <none>
I20260620 06:07:01.549858 14786 fs_manager.cc:359] Metadata directory not provided
I20260620 06:07:01.549913 14786 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:07:01.550063 14786 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 06:07:01.551995 14786 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781935603120653-11672-0/minicluster-data/ts-0/data/instance:
uuid: "23ec863b162146b59377a44ae659a163"
format_stamp: "Formatted at 2026-06-20 06:07:01 on dist-test-slave-9hm5"
server_key: "90db007ec08e7a41e49ee82e745bd589"
server_key_iv: "3c092a9d9590e29ea90ac5748712f409"
server_key_version: "encryptionkey@0"
I20260620 06:07:01.552505 14786 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781935603120653-11672-0/minicluster-data/ts-0/wal/instance:
uuid: "23ec863b162146b59377a44ae659a163"
format_stamp: "Formatted at 2026-06-20 06:07:01 on dist-test-slave-9hm5"
server_key: "90db007ec08e7a41e49ee82e745bd589"
server_key_iv: "3c092a9d9590e29ea90ac5748712f409"
server_key_version: "encryptionkey@0"
I20260620 06:07:01.555998 14786 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 06:07:01.558224 14801 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:07:01.559371 14786 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 06:07:01.559484 14786 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781935603120653-11672-0/minicluster-data/ts-0/data,/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781935603120653-11672-0/minicluster-data/ts-0/wal
uuid: "23ec863b162146b59377a44ae659a163"
format_stamp: "Formatted at 2026-06-20 06:07:01 on dist-test-slave-9hm5"
server_key: "90db007ec08e7a41e49ee82e745bd589"
server_key_iv: "3c092a9d9590e29ea90ac5748712f409"
server_key_version: "encryptionkey@0"
I20260620 06:07:01.559580 14786 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781935603120653-11672-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781935603120653-11672-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781935603120653-11672-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:07:01.595220 14786 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:07:01.599009 14786 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:07:01.599215 14786 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:07:01.599874 14786 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 06:07:01.600672 14786 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 06:07:01.600728 14786 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 06:07:01.600811 14786 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 06:07:01.600844 14786 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 06:07:01.609375 14786 rpc_server.cc:307] RPC server started. Bound to: 127.11.102.1:45595
I20260620 06:07:01.609397 14914 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.102.1:45595 every 8 connection(s)
I20260620 06:07:01.610271 14786 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781935603120653-11672-0/minicluster-data/ts-0/data/info.pb
I20260620 06:07:01.611284 11672 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9HMp43/build/debug/bin/kudu as pid 14786
I20260620 06:07:01.611413 11672 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781935603120653-11672-0/minicluster-data/ts-0/wal/instance
I20260620 06:07:01.611766 11672 external_mini_cluster.cc:1428] Setting key baf12a54eaa4506bceb4c2045e71ffa3
Jun 20 06:07:01 dist-test-slave-9hm5 krb5kdc[14679](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935621, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.102.1@KRBTEST.COM for kudu/127.11.102.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.11.102.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.102.2@KRBTEST.COM" created.
I20260620 06:07:01.625186 14763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:07:01.612227 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.102.1:33445 (local address 127.11.102.62:41431)
0620 06:07:01.612379 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0620 06:07:01.612384 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 06:07:01.613415 (+ 1031us) server_negotiation.cc:407] Connection header received
0620 06:07:01.615340 (+ 1925us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:07:01.615345 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:07:01.615406 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:07:01.615503 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:07:01.616706 (+ 1203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:07:01.617605 (+ 899us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:07:01.618708 (+ 1103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:07:01.619133 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:07:01.619251 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:07:01.621799 (+ 2548us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:07:01.621802 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:07:01.621803 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:07:01.621826 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:07:01.623107 (+ 1281us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:07:01.623555 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:07:01.623557 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:07:01.623558 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:07:01.623713 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:07:01.624080 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:07:01.624082 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:07:01.624083 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:07:01.624241 (+ 158us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:07:01.624317 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0620 06:07:01.624938 (+ 621us) server_negotiation.cc:299] Negotiation successful
0620 06:07:01.625030 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260620 06:07:01.626099 14917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:07:01.612676 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.102.62:41431 (local address 127.11.102.1:33445)
0620 06:07:01.613252 (+ 576us) negotiation.cc:107] Waiting for socket to connect
0620 06:07:01.613309 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0620 06:07:01.615142 (+ 1833us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:07:01.615703 (+ 561us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:07:01.615715 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:07:01.616070 (+ 355us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:07:01.616542 (+ 472us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:07:01.616555 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:07:01.617725 (+ 1170us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:07:01.617727 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:07:01.618583 (+ 856us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:07:01.618591 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:07:01.619272 (+ 681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:07:01.619275 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:07:01.619405 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:07:01.619993 (+ 588us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:07:01.620009 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:07:01.621643 (+ 1634us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:07:01.623220 (+ 1577us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:07:01.623225 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:07:01.623235 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:07:01.623466 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:07:01.623819 (+ 353us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:07:01.623822 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:07:01.623824 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:07:01.623963 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:07:01.624345 (+ 382us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:07:01.624355 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:07:01.624675 (+ 320us) client_negotiation.cc:769] Sending connection context
0620 06:07:01.624908 (+ 233us) client_negotiation.cc:240] Negotiation successful
0620 06:07:01.625264 (+ 356us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":326,"thread_start_us":137,"threads_started":1}
I20260620 06:07:01.627374 14915 heartbeater.cc:347] Connected to a master server at 127.11.102.62:41431
I20260620 06:07:01.627667 14915 heartbeater.cc:464] Registering TS with master...
I20260620 06:07:01.628258 14915 heartbeater.cc:511] Master 127.11.102.62:41431 requested a full tablet report, sending...
I20260620 06:07:01.629997 14724 ts_manager.cc:194] Registered new tserver with Master: 23ec863b162146b59377a44ae659a163 (127.11.102.1:45595)
I20260620 06:07:01.631628 14724 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.102.1@KRBTEST.COM'} at 127.11.102.1:33445
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.102.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781935603120653-11672-0/krb5kdc/kudu_127.11.102.2.keytab.
Entry for principal kudu/127.11.102.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781935603120653-11672-0/krb5kdc/kudu_127.11.102.2.keytab.
I20260620 06:07:01.638249 11672 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9HMp43/build/debug/bin/kudu
/tmp/dist-test-task9HMp43/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781935603120653-11672-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781935603120653-11672-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781935603120653-11672-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781935603120653-11672-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.102.2:0
--local_ip_for_outbound_sockets=127.11.102.2
--webserver_interface=127.11.102.2
--webserver_port=0
--tserver_master_addrs=127.11.102.62:41431
--keytab_file=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781935603120653-11672-0/krb5kdc/kudu_127.11.102.2.keytab
--principal=kudu/127.11.102.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.102.20:45769
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781935603120653-11672-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781935603120653-11672-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781935603120653-11672-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 06:07:01.748289 14920 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:07:01.748528 14920 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:07:01.748600 14920 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:07:01.752573 14920 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:07:01.752624 14920 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:07:01.752707 14920 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.102.2
I20260620 06:07:01.757790 14920 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.102.20:45769
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781935603120653-11672-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781935603120653-11672-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781935603120653-11672-0/krb5kdc/kudu_127.11.102.2.keytab
--principal=kudu/127.11.102.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.102.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781935603120653-11672-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.11.102.2
--webserver_port=0
--tserver_master_addrs=127.11.102.62:41431
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14920
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.102.2
--log_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781935603120653-11672-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 05:28:48 UTC on ccycloud.quasar-fixhms.root.comops.site
build id 4457020
I20260620 06:07:01.758755 14920 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:07:01.759781 14920 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 06:07:01.768941 14926 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:07:01.768993 14925 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:07:01.769091 14928 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:07:01.768980 14920 server_base.cc:1034] running on GCE node
I20260620 06:07:01.769582 14920 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:07:01.770161 14920 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:07:01.771338 14920 hybrid_clock.cc:648] HybridClock initialized: now 1781935621771322 us; error 28 us; skew 500 ppm
Jun 20 06:07:01 dist-test-slave-9hm5 krb5kdc[14679](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935621, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.102.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 06:07:01.773953 14920 init.cc:377] Logged in from keytab as kudu/127.11.102.2@KRBTEST.COM (short username kudu)
I20260620 06:07:01.774765 14920 webserver.cc:460] Webserver started at http://127.11.102.2:45661/ using document root <none> and password file <none>
I20260620 06:07:01.775262 14920 fs_manager.cc:359] Metadata directory not provided
I20260620 06:07:01.775317 14920 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:07:01.775470 14920 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 06:07:01.777402 14920 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781935603120653-11672-0/minicluster-data/ts-1/data/instance:
uuid: "c9979b175de8414589125aa4343d5d79"
format_stamp: "Formatted at 2026-06-20 06:07:01 on dist-test-slave-9hm5"
server_key: "2f7089eba65224c90c65e02ba5f25866"
server_key_iv: "7383ba2922716e1d52805200ed2fdd81"
server_key_version: "encryptionkey@0"
I20260620 06:07:01.777887 14920 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781935603120653-11672-0/minicluster-data/ts-1/wal/instance:
uuid: "c9979b175de8414589125aa4343d5d79"
format_stamp: "Formatted at 2026-06-20 06:07:01 on dist-test-slave-9hm5"
server_key: "2f7089eba65224c90c65e02ba5f25866"
server_key_iv: "7383ba2922716e1d52805200ed2fdd81"
server_key_version: "encryptionkey@0"
I20260620 06:07:01.781456 14920 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260620 06:07:01.783685 14935 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:07:01.784865 14920 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 06:07:01.784952 14920 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781935603120653-11672-0/minicluster-data/ts-1/data,/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781935603120653-11672-0/minicluster-data/ts-1/wal
uuid: "c9979b175de8414589125aa4343d5d79"
format_stamp: "Formatted at 2026-06-20 06:07:01 on dist-test-slave-9hm5"
server_key: "2f7089eba65224c90c65e02ba5f25866"
server_key_iv: "7383ba2922716e1d52805200ed2fdd81"
server_key_version: "encryptionkey@0"
I20260620 06:07:01.785028 14920 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781935603120653-11672-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781935603120653-11672-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781935603120653-11672-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:07:01.800400 14920 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:07:01.802938 14920 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:07:01.803076 14920 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:07:01.803557 14920 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 06:07:01.804358 14920 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 06:07:01.804430 14920 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 06:07:01.804502 14920 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 06:07:01.804554 14920 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 06:07:01.813041 14920 rpc_server.cc:307] RPC server started. Bound to: 127.11.102.2:33543
I20260620 06:07:01.813064 15048 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.102.2:33543 every 8 connection(s)
I20260620 06:07:01.813866 14920 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781935603120653-11672-0/minicluster-data/ts-1/data/info.pb
Jun 20 06:07:01 dist-test-slave-9hm5 krb5kdc[14679](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935621, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.102.2@KRBTEST.COM for kudu/127.11.102.62@KRBTEST.COM
I20260620 06:07:01.824105 11672 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9HMp43/build/debug/bin/kudu as pid 14920
I20260620 06:07:01.824244 11672 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781935603120653-11672-0/minicluster-data/ts-1/wal/instance
I20260620 06:07:01.824537 11672 external_mini_cluster.cc:1428] Setting key 055aa3c18c780ee3264fca018fd8724c
I20260620 06:07:01.827634 14763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:07:01.815568 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.102.2:36783 (local address 127.11.102.62:41431)
0620 06:07:01.815716 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0620 06:07:01.815720 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 06:07:01.816553 (+ 833us) server_negotiation.cc:407] Connection header received
0620 06:07:01.817866 (+ 1313us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:07:01.817870 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:07:01.817927 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:07:01.818030 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:07:01.819193 (+ 1163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:07:01.819788 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:07:01.820955 (+ 1167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:07:01.821185 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:07:01.821297 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:07:01.823761 (+ 2464us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:07:01.823764 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:07:01.823766 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:07:01.823788 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:07:01.825382 (+ 1594us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:07:01.825983 (+ 601us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:07:01.825986 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:07:01.825988 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:07:01.826074 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:07:01.826408 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:07:01.826411 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:07:01.826414 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:07:01.826670 (+ 256us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:07:01.826775 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0620 06:07:01.827214 (+ 439us) server_negotiation.cc:299] Negotiation successful
0620 06:07:01.827350 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260620 06:07:01.828419 15051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:07:01.815983 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.102.62:41431 (local address 127.11.102.2:36783)
0620 06:07:01.816414 (+ 431us) negotiation.cc:107] Waiting for socket to connect
0620 06:07:01.816454 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0620 06:07:01.817663 (+ 1209us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:07:01.818193 (+ 530us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:07:01.818200 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:07:01.818539 (+ 339us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:07:01.819047 (+ 508us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:07:01.819061 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:07:01.819922 (+ 861us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:07:01.819925 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:07:01.820829 (+ 904us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:07:01.820838 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:07:01.821331 (+ 493us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:07:01.821335 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:07:01.821498 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:07:01.822086 (+ 588us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:07:01.822101 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:07:01.823528 (+ 1427us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:07:01.825489 (+ 1961us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:07:01.825494 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:07:01.825509 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:07:01.825864 (+ 355us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:07:01.826173 (+ 309us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:07:01.826175 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:07:01.826176 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:07:01.826310 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:07:01.826769 (+ 459us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:07:01.826774 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:07:01.826985 (+ 211us) client_negotiation.cc:769] Sending connection context
0620 06:07:01.827135 (+ 150us) client_negotiation.cc:240] Negotiation successful
0620 06:07:01.827378 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":257,"thread_start_us":117,"threads_started":1}
I20260620 06:07:01.829528 15049 heartbeater.cc:347] Connected to a master server at 127.11.102.62:41431
I20260620 06:07:01.829763 15049 heartbeater.cc:464] Registering TS with master...
I20260620 06:07:01.830331 15049 heartbeater.cc:511] Master 127.11.102.62:41431 requested a full tablet report, sending...
I20260620 06:07:01.831357 14724 ts_manager.cc:194] Registered new tserver with Master: c9979b175de8414589125aa4343d5d79 (127.11.102.2:33543)
I20260620 06:07:01.831950 14724 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.102.2@KRBTEST.COM'} at 127.11.102.2:36783
WARNING: no policy specified for kudu/127.11.102.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.102.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.102.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781935603120653-11672-0/krb5kdc/kudu_127.11.102.3.keytab.
Entry for principal kudu/127.11.102.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781935603120653-11672-0/krb5kdc/kudu_127.11.102.3.keytab.
I20260620 06:07:01.850663 11672 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9HMp43/build/debug/bin/kudu
/tmp/dist-test-task9HMp43/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781935603120653-11672-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781935603120653-11672-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781935603120653-11672-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781935603120653-11672-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.102.3:0
--local_ip_for_outbound_sockets=127.11.102.3
--webserver_interface=127.11.102.3
--webserver_port=0
--tserver_master_addrs=127.11.102.62:41431
--keytab_file=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781935603120653-11672-0/krb5kdc/kudu_127.11.102.3.keytab
--principal=kudu/127.11.102.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.102.20:45769
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781935603120653-11672-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781935603120653-11672-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781935603120653-11672-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 06:07:01.958712 15054 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:07:01.958938 15054 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:07:01.958977 15054 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:07:01.962919 15054 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:07:01.962970 15054 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:07:01.963025 15054 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.102.3
I20260620 06:07:01.968036 15054 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.102.20:45769
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781935603120653-11672-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781935603120653-11672-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781935603120653-11672-0/krb5kdc/kudu_127.11.102.3.keytab
--principal=kudu/127.11.102.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.102.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781935603120653-11672-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.11.102.3
--webserver_port=0
--tserver_master_addrs=127.11.102.62:41431
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15054
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.102.3
--log_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781935603120653-11672-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 05:28:48 UTC on ccycloud.quasar-fixhms.root.comops.site
build id 4457020
I20260620 06:07:01.968900 15054 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:07:01.969790 15054 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 06:07:01.978808 15062 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:07:01.978870 15054 server_base.cc:1034] running on GCE node
W20260620 06:07:01.978832 15060 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:07:01.978808 15059 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:07:01.979676 15054 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:07:01.980265 15054 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:07:01.981468 15054 hybrid_clock.cc:648] HybridClock initialized: now 1781935621981456 us; error 23 us; skew 500 ppm
Jun 20 06:07:01 dist-test-slave-9hm5 krb5kdc[14679](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935621, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.102.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 06:07:01.984179 15054 init.cc:377] Logged in from keytab as kudu/127.11.102.3@KRBTEST.COM (short username kudu)
I20260620 06:07:01.985016 15054 webserver.cc:460] Webserver started at http://127.11.102.3:42729/ using document root <none> and password file <none>
I20260620 06:07:01.985561 15054 fs_manager.cc:359] Metadata directory not provided
I20260620 06:07:01.985651 15054 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:07:01.985873 15054 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 06:07:01.987920 15054 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781935603120653-11672-0/minicluster-data/ts-2/data/instance:
uuid: "87073426d53e46cf8bed20f66d7b6519"
format_stamp: "Formatted at 2026-06-20 06:07:01 on dist-test-slave-9hm5"
server_key: "db4a20700d10e42a5f244b63424313e6"
server_key_iv: "75947d0e738b85c5ffd0a2898ddb3f3f"
server_key_version: "encryptionkey@0"
I20260620 06:07:01.988520 15054 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781935603120653-11672-0/minicluster-data/ts-2/wal/instance:
uuid: "87073426d53e46cf8bed20f66d7b6519"
format_stamp: "Formatted at 2026-06-20 06:07:01 on dist-test-slave-9hm5"
server_key: "db4a20700d10e42a5f244b63424313e6"
server_key_iv: "75947d0e738b85c5ffd0a2898ddb3f3f"
server_key_version: "encryptionkey@0"
I20260620 06:07:01.992321 15054 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260620 06:07:01.994788 15069 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:07:01.996119 15054 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 06:07:01.996237 15054 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781935603120653-11672-0/minicluster-data/ts-2/data,/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781935603120653-11672-0/minicluster-data/ts-2/wal
uuid: "87073426d53e46cf8bed20f66d7b6519"
format_stamp: "Formatted at 2026-06-20 06:07:01 on dist-test-slave-9hm5"
server_key: "db4a20700d10e42a5f244b63424313e6"
server_key_iv: "75947d0e738b85c5ffd0a2898ddb3f3f"
server_key_version: "encryptionkey@0"
I20260620 06:07:01.996335 15054 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781935603120653-11672-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781935603120653-11672-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781935603120653-11672-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:07:02.020278 15054 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:07:02.023510 15054 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:07:02.023731 15054 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:07:02.024339 15054 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 06:07:02.025336 15054 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 06:07:02.025398 15054 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 06:07:02.025466 15054 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 06:07:02.025497 15054 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 06:07:02.033871 15054 rpc_server.cc:307] RPC server started. Bound to: 127.11.102.3:39997
I20260620 06:07:02.033902 15182 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.102.3:39997 every 8 connection(s)
I20260620 06:07:02.034744 15054 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781935603120653-11672-0/minicluster-data/ts-2/data/info.pb
I20260620 06:07:02.036164 11672 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9HMp43/build/debug/bin/kudu as pid 15054
I20260620 06:07:02.036286 11672 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781935603120653-11672-0/minicluster-data/ts-2/wal/instance
I20260620 06:07:02.036604 11672 external_mini_cluster.cc:1428] Setting key f1600a5a273ace00750e6149686939cc
Jun 20 06:07:02 dist-test-slave-9hm5 krb5kdc[14679](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935621, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.102.3@KRBTEST.COM for kudu/127.11.102.62@KRBTEST.COM
I20260620 06:07:02.048497 14763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:07:02.036703 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.102.3:47925 (local address 127.11.102.62:41431)
0620 06:07:02.036824 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0620 06:07:02.036827 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 06:07:02.037819 (+ 992us) server_negotiation.cc:407] Connection header received
0620 06:07:02.039493 (+ 1674us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:07:02.039496 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:07:02.039541 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:07:02.039659 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:07:02.040785 (+ 1126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:07:02.041305 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:07:02.042461 (+ 1156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:07:02.042683 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:07:02.042770 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:07:02.045133 (+ 2363us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:07:02.045136 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:07:02.045138 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:07:02.045161 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:07:02.046645 (+ 1484us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:07:02.047098 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:07:02.047101 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:07:02.047102 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:07:02.047174 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:07:02.047464 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:07:02.047467 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:07:02.047468 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:07:02.047726 (+ 258us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:07:02.047805 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0620 06:07:02.048278 (+ 473us) server_negotiation.cc:299] Negotiation successful
0620 06:07:02.048377 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260620 06:07:02.049212 15185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:07:02.037135 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.102.62:41431 (local address 127.11.102.3:47925)
0620 06:07:02.037673 (+ 538us) negotiation.cc:107] Waiting for socket to connect
0620 06:07:02.037714 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0620 06:07:02.039289 (+ 1575us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:07:02.039797 (+ 508us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:07:02.039804 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:07:02.040155 (+ 351us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:07:02.040667 (+ 512us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:07:02.040681 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:07:02.041455 (+ 774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:07:02.041458 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:07:02.042335 (+ 877us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:07:02.042344 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:07:02.042811 (+ 467us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:07:02.042813 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:07:02.042956 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:07:02.043481 (+ 525us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:07:02.043496 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:07:02.044990 (+ 1494us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:07:02.046749 (+ 1759us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:07:02.046753 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:07:02.046764 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:07:02.047011 (+ 247us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:07:02.047250 (+ 239us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:07:02.047252 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:07:02.047253 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:07:02.047371 (+ 118us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:07:02.047816 (+ 445us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:07:02.047821 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:07:02.048042 (+ 221us) client_negotiation.cc:769] Sending connection context
0620 06:07:02.048194 (+ 152us) client_negotiation.cc:240] Negotiation successful
0620 06:07:02.048425 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":330,"thread_start_us":158,"threads_started":1}
I20260620 06:07:02.050217 15183 heartbeater.cc:347] Connected to a master server at 127.11.102.62:41431
I20260620 06:07:02.050438 15183 heartbeater.cc:464] Registering TS with master...
I20260620 06:07:02.050896 15183 heartbeater.cc:511] Master 127.11.102.62:41431 requested a full tablet report, sending...
I20260620 06:07:02.051852 14724 ts_manager.cc:194] Registered new tserver with Master: 87073426d53e46cf8bed20f66d7b6519 (127.11.102.3:39997)
I20260620 06:07:02.052418 14724 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.102.3@KRBTEST.COM'} at 127.11.102.3:47925
I20260620 06:07:02.062201 11672 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 20 06:07:02 dist-test-slave-9hm5 krb5kdc[14679](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935622, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Jun 20 06:07:02 dist-test-slave-9hm5 krb5kdc[14679](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935622, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.11.102.62@KRBTEST.COM
I20260620 06:07:02.087890 14763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:07:02.079345 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54344 (local address 127.11.102.62:41431)
0620 06:07:02.079535 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0620 06:07:02.079540 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 06:07:02.079722 (+ 182us) server_negotiation.cc:407] Connection header received
0620 06:07:02.079917 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:07:02.079921 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:07:02.079967 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:07:02.080058 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:07:02.080741 (+ 683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:07:02.081266 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:07:02.082119 (+ 853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:07:02.082339 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:07:02.082454 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:07:02.084860 (+ 2406us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:07:02.084863 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:07:02.084865 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:07:02.084897 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:07:02.086176 (+ 1279us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:07:02.086599 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:07:02.086602 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:07:02.086603 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:07:02.086688 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:07:02.086962 (+ 274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:07:02.086965 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:07:02.086966 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:07:02.087173 (+ 207us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:07:02.087282 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0620 06:07:02.087531 (+ 249us) server_negotiation.cc:299] Negotiation successful
0620 06:07:02.087690 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
W20260620 06:07:02.088337 14724 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:54344
I20260620 06:07:02.089335 11672 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9HMp43/build/debug/bin/kudu with pid 14786
I20260620 06:07:02.095222 11672 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9HMp43/build/debug/bin/kudu with pid 14920
I20260620 06:07:02.100411 11672 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9HMp43/build/debug/bin/kudu with pid 15054
I20260620 06:07:02.105285 11672 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9HMp43/build/debug/bin/kudu with pid 14692
2026-06-20T06:07:02Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (2107 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781935603120653-11672-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 06:07:02 dist-test-slave-9hm5 krb5kdc[15195](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 06:07:02 dist-test-slave-9hm5 krb5kdc[15195](info): set up 2 sockets
Jun 20 06:07:02 dist-test-slave-9hm5 krb5kdc[15195](info): commencing operation
krb5kdc: starting...
W20260620 06:07:03.147644 11672 mini_kdc.cc:121] Time spent starting KDC: real 1.021s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781935603120653-11672-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781935603120653-11672-0/krb5kdc/test-user.keytab.
Jun 20 06:07:03 dist-test-slave-9hm5 krb5kdc[15195](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935623, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T06:07:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T06:07:03Z Disabled control of system clock
WARNING: no policy specified for kudu/127.11.102.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.102.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.102.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781935603120653-11672-0/krb5kdc/kudu_127.11.102.62.keytab.
Entry for principal kudu/127.11.102.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781935603120653-11672-0/krb5kdc/kudu_127.11.102.62.keytab.
I20260620 06:07:03.266582 11672 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9HMp43/build/debug/bin/kudu
/tmp/dist-test-task9HMp43/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781935603120653-11672-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781935603120653-11672-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781935603120653-11672-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781935603120653-11672-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.102.62:43573
--webserver_interface=127.11.102.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781935603120653-11672-0/krb5kdc/kudu_127.11.102.62.keytab
--principal=kudu/127.11.102.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.102.20:35239
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.102.62:43573
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781935603120653-11672-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781935603120653-11672-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781935603120653-11672-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 06:07:03.372010 15208 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:07:03.372258 15208 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:07:03.372298 15208 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:07:03.376272 15208 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 06:07:03.376318 15208 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:07:03.376338 15208 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:07:03.376394 15208 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 06:07:03.376423 15208 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 06:07:03.381503 15208 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.102.20:35239
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781935603120653-11672-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781935603120653-11672-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.102.62:43573
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781935603120653-11672-0/krb5kdc/kudu_127.11.102.62.keytab
--principal=kudu/127.11.102.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.102.62:43573
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781935603120653-11672-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.102.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15208
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781935603120653-11672-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 05:28:48 UTC on ccycloud.quasar-fixhms.root.comops.site
build id 4457020
I20260620 06:07:03.382450 15208 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:07:03.383400 15208 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 06:07:03.391441 15214 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:07:03.391462 15213 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:07:03.391547 15208 server_base.cc:1034] running on GCE node
W20260620 06:07:03.391759 15216 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:07:03.392279 15208 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:07:03.393323 15208 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:07:03.394496 15208 hybrid_clock.cc:648] HybridClock initialized: now 1781935623394462 us; error 45 us; skew 500 ppm
Jun 20 06:07:03 dist-test-slave-9hm5 krb5kdc[15195](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935623, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.102.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 06:07:03.397166 15208 init.cc:377] Logged in from keytab as kudu/127.11.102.62@KRBTEST.COM (short username kudu)
I20260620 06:07:03.398036 15208 webserver.cc:460] Webserver started at http://127.11.102.62:35553/ using document root <none> and password file <none>
I20260620 06:07:03.398576 15208 fs_manager.cc:359] Metadata directory not provided
I20260620 06:07:03.398662 15208 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:07:03.398885 15208 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 06:07:03.400858 15208 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781935603120653-11672-0/minicluster-data/master-0/data/instance:
uuid: "8117af67701c4eef8c01063b23104438"
format_stamp: "Formatted at 2026-06-20 06:07:03 on dist-test-slave-9hm5"
server_key: "be44712f8c6dec5caf79713bb9436af6"
server_key_iv: "77a1633dd094d511b35756d7cf0435b8"
server_key_version: "encryptionkey@0"
I20260620 06:07:03.401420 15208 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781935603120653-11672-0/minicluster-data/master-0/wal/instance:
uuid: "8117af67701c4eef8c01063b23104438"
format_stamp: "Formatted at 2026-06-20 06:07:03 on dist-test-slave-9hm5"
server_key: "be44712f8c6dec5caf79713bb9436af6"
server_key_iv: "77a1633dd094d511b35756d7cf0435b8"
server_key_version: "encryptionkey@0"
I20260620 06:07:03.404923 15208 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 06:07:03.407249 15223 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:07:03.408481 15208 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 06:07:03.408628 15208 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781935603120653-11672-0/minicluster-data/master-0/data,/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781935603120653-11672-0/minicluster-data/master-0/wal
uuid: "8117af67701c4eef8c01063b23104438"
format_stamp: "Formatted at 2026-06-20 06:07:03 on dist-test-slave-9hm5"
server_key: "be44712f8c6dec5caf79713bb9436af6"
server_key_iv: "77a1633dd094d511b35756d7cf0435b8"
server_key_version: "encryptionkey@0"
I20260620 06:07:03.408740 15208 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781935603120653-11672-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781935603120653-11672-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781935603120653-11672-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:07:03.430593 15208 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:07:03.433482 15208 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:07:03.433689 15208 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:07:03.441135 15208 rpc_server.cc:307] RPC server started. Bound to: 127.11.102.62:43573
I20260620 06:07:03.441164 15275 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.102.62:43573 every 8 connection(s)
I20260620 06:07:03.442066 15208 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781935603120653-11672-0/minicluster-data/master-0/data/info.pb
I20260620 06:07:03.442421 11672 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9HMp43/build/debug/bin/kudu as pid 15208
I20260620 06:07:03.442572 11672 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781935603120653-11672-0/minicluster-data/master-0/wal/instance
I20260620 06:07:03.442876 11672 external_mini_cluster.cc:1428] Setting key 946e5b05a647c67685535b11936940dc
I20260620 06:07:03.445314 15276 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 06:07:03.451838 15276 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8117af67701c4eef8c01063b23104438: Bootstrap starting.
Jun 20 06:07:03 dist-test-slave-9hm5 krb5kdc[15195](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935623, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.102.62@KRBTEST.COM
I20260620 06:07:03.454540 15276 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8117af67701c4eef8c01063b23104438: Neither blocks nor log segments found. Creating new log.
I20260620 06:07:03.455451 15276 log.cc:824] T 00000000000000000000000000000000 P 8117af67701c4eef8c01063b23104438: Log is configured to *not* fsync() on all Append() calls
I20260620 06:07:03.457635 15276 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8117af67701c4eef8c01063b23104438: No bootstrap required, opened a new log
I20260620 06:07:03.458056 15279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:07:03.444241 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42482 (local address 127.11.102.62:43573)
0620 06:07:03.444641 (+ 400us) server_negotiation.cc:206] Beginning negotiation
0620 06:07:03.444659 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0620 06:07:03.444701 (+ 42us) server_negotiation.cc:407] Connection header received
0620 06:07:03.445545 (+ 844us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:07:03.445566 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:07:03.445956 (+ 390us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:07:03.446315 (+ 359us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:07:03.446964 (+ 649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:07:03.448478 (+ 1514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:07:03.449294 (+ 816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:07:03.449952 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:07:03.450119 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:07:03.452671 (+ 2552us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:07:03.452677 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:07:03.452692 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:07:03.452722 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:07:03.454723 (+ 2001us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:07:03.455154 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:07:03.455167 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:07:03.455172 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:07:03.455318 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:07:03.455690 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:07:03.455697 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:07:03.455701 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:07:03.456222 (+ 521us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:07:03.456408 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0620 06:07:03.456725 (+ 317us) server_negotiation.cc:299] Negotiation successful
0620 06:07:03.456992 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":266,"thread_start_us":105,"threads_started":1}
I20260620 06:07:03.461596 15276 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8117af67701c4eef8c01063b23104438 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8117af67701c4eef8c01063b23104438" member_type: VOTER last_known_addr { host: "127.11.102.62" port: 43573 } }
I20260620 06:07:03.461889 15276 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8117af67701c4eef8c01063b23104438 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 06:07:03.461956 15276 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8117af67701c4eef8c01063b23104438 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8117af67701c4eef8c01063b23104438, State: Initialized, Role: FOLLOWER
I20260620 06:07:03.462385 15276 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8117af67701c4eef8c01063b23104438 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8117af67701c4eef8c01063b23104438" member_type: VOTER last_known_addr { host: "127.11.102.62" port: 43573 } }
I20260620 06:07:03.462507 15276 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8117af67701c4eef8c01063b23104438 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 06:07:03.462589 15276 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8117af67701c4eef8c01063b23104438 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 06:07:03.462689 15276 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8117af67701c4eef8c01063b23104438 [term 0 FOLLOWER]: Advancing to term 1
I20260620 06:07:03.463567 15276 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8117af67701c4eef8c01063b23104438 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8117af67701c4eef8c01063b23104438" member_type: VOTER last_known_addr { host: "127.11.102.62" port: 43573 } }
I20260620 06:07:03.463955 15276 leader_election.cc:302] T 00000000000000000000000000000000 P 8117af67701c4eef8c01063b23104438 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8117af67701c4eef8c01063b23104438; no voters:
I20260620 06:07:03.464249 15276 leader_election.cc:288] T 00000000000000000000000000000000 P 8117af67701c4eef8c01063b23104438 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 06:07:03.464370 15281 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8117af67701c4eef8c01063b23104438 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 06:07:03.464612 15281 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8117af67701c4eef8c01063b23104438 [term 1 LEADER]: Becoming Leader. State: Replica: 8117af67701c4eef8c01063b23104438, State: Running, Role: LEADER
I20260620 06:07:03.464936 15281 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8117af67701c4eef8c01063b23104438 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8117af67701c4eef8c01063b23104438" member_type: VOTER last_known_addr { host: "127.11.102.62" port: 43573 } }
I20260620 06:07:03.465415 15276 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8117af67701c4eef8c01063b23104438 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 06:07:03.466465 15283 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8117af67701c4eef8c01063b23104438 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8117af67701c4eef8c01063b23104438. Latest consensus state: current_term: 1 leader_uuid: "8117af67701c4eef8c01063b23104438" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8117af67701c4eef8c01063b23104438" member_type: VOTER last_known_addr { host: "127.11.102.62" port: 43573 } } }
I20260620 06:07:03.466598 15283 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8117af67701c4eef8c01063b23104438 [sys.catalog]: This master's current role is: LEADER
I20260620 06:07:03.466861 15282 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8117af67701c4eef8c01063b23104438 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8117af67701c4eef8c01063b23104438" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8117af67701c4eef8c01063b23104438" member_type: VOTER last_known_addr { host: "127.11.102.62" port: 43573 } } }
I20260620 06:07:03.466952 15282 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8117af67701c4eef8c01063b23104438 [sys.catalog]: This master's current role is: LEADER
I20260620 06:07:03.466950 15290 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 06:07:03.470005 15290 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 06:07:03.474982 15290 catalog_manager.cc:1347] Generated new cluster ID: 2bcc179c99a44f6ca993c6d964e6ed2e
I20260620 06:07:03.475052 15290 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 06:07:03.486948 15290 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 06:07:03.487929 15290 catalog_manager.cc:1504] Loading token signing keys...
I20260620 06:07:03.496284 15290 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8117af67701c4eef8c01063b23104438: Generated new TSK 0
I20260620 06:07:03.497134 15290 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.11.102.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.102.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.102.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781935603120653-11672-0/krb5kdc/kudu_127.11.102.1.keytab.
Entry for principal kudu/127.11.102.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781935603120653-11672-0/krb5kdc/kudu_127.11.102.1.keytab.
I20260620 06:07:03.526726 11672 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9HMp43/build/debug/bin/kudu
/tmp/dist-test-task9HMp43/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781935603120653-11672-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781935603120653-11672-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781935603120653-11672-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781935603120653-11672-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.102.1:0
--local_ip_for_outbound_sockets=127.11.102.1
--webserver_interface=127.11.102.1
--webserver_port=0
--tserver_master_addrs=127.11.102.62:43573
--keytab_file=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781935603120653-11672-0/krb5kdc/kudu_127.11.102.1.keytab
--principal=kudu/127.11.102.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.102.20:35239
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781935603120653-11672-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781935603120653-11672-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781935603120653-11672-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 06:07:03.637208 15302 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:07:03.637478 15302 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:07:03.637530 15302 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:07:03.641671 15302 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:07:03.641732 15302 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:07:03.641793 15302 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.102.1
I20260620 06:07:03.649928 15302 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.102.20:35239
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781935603120653-11672-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781935603120653-11672-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781935603120653-11672-0/krb5kdc/kudu_127.11.102.1.keytab
--principal=kudu/127.11.102.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.102.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781935603120653-11672-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.11.102.1
--webserver_port=0
--tserver_master_addrs=127.11.102.62:43573
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15302
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.102.1
--log_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781935603120653-11672-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 05:28:48 UTC on ccycloud.quasar-fixhms.root.comops.site
build id 4457020
I20260620 06:07:03.651360 15302 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:07:03.652763 15302 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 06:07:03.662161 15307 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:07:03.662253 15302 server_base.cc:1034] running on GCE node
W20260620 06:07:03.662228 15308 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:07:03.662424 15310 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:07:03.662820 15302 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:07:03.663441 15302 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:07:03.664613 15302 hybrid_clock.cc:648] HybridClock initialized: now 1781935623664633 us; error 63 us; skew 500 ppm
Jun 20 06:07:03 dist-test-slave-9hm5 krb5kdc[15195](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935623, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.102.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 06:07:03.667702 15302 init.cc:377] Logged in from keytab as kudu/127.11.102.1@KRBTEST.COM (short username kudu)
I20260620 06:07:03.668593 15302 webserver.cc:460] Webserver started at http://127.11.102.1:32809/ using document root <none> and password file <none>
I20260620 06:07:03.669127 15302 fs_manager.cc:359] Metadata directory not provided
I20260620 06:07:03.669224 15302 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:07:03.669437 15302 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 06:07:03.671360 15302 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781935603120653-11672-0/minicluster-data/ts-0/data/instance:
uuid: "9198dc22c7654a29946a3c9b9d09fcff"
format_stamp: "Formatted at 2026-06-20 06:07:03 on dist-test-slave-9hm5"
server_key: "fdd2db01ce0723cd7945efc281c6bf5c"
server_key_iv: "429a2a38bd24f4724dc833699bdec01e"
server_key_version: "encryptionkey@0"
I20260620 06:07:03.671924 15302 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781935603120653-11672-0/minicluster-data/ts-0/wal/instance:
uuid: "9198dc22c7654a29946a3c9b9d09fcff"
format_stamp: "Formatted at 2026-06-20 06:07:03 on dist-test-slave-9hm5"
server_key: "fdd2db01ce0723cd7945efc281c6bf5c"
server_key_iv: "429a2a38bd24f4724dc833699bdec01e"
server_key_version: "encryptionkey@0"
I20260620 06:07:03.675432 15302 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260620 06:07:03.677721 15317 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:07:03.678830 15302 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 06:07:03.678956 15302 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781935603120653-11672-0/minicluster-data/ts-0/data,/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781935603120653-11672-0/minicluster-data/ts-0/wal
uuid: "9198dc22c7654a29946a3c9b9d09fcff"
format_stamp: "Formatted at 2026-06-20 06:07:03 on dist-test-slave-9hm5"
server_key: "fdd2db01ce0723cd7945efc281c6bf5c"
server_key_iv: "429a2a38bd24f4724dc833699bdec01e"
server_key_version: "encryptionkey@0"
I20260620 06:07:03.679059 15302 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781935603120653-11672-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781935603120653-11672-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781935603120653-11672-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:07:03.698263 15302 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:07:03.701491 15302 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:07:03.701687 15302 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:07:03.702234 15302 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 06:07:03.703017 15302 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 06:07:03.703094 15302 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 06:07:03.703152 15302 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 06:07:03.703209 15302 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 06:07:03.712179 15302 rpc_server.cc:307] RPC server started. Bound to: 127.11.102.1:45403
I20260620 06:07:03.712241 15430 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.102.1:45403 every 8 connection(s)
I20260620 06:07:03.713145 15302 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781935603120653-11672-0/minicluster-data/ts-0/data/info.pb
I20260620 06:07:03.713481 11672 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9HMp43/build/debug/bin/kudu as pid 15302
I20260620 06:07:03.713634 11672 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781935603120653-11672-0/minicluster-data/ts-0/wal/instance
I20260620 06:07:03.713951 11672 external_mini_cluster.cc:1428] Setting key d7f8f12be42d09e7536fc5e8abec9576
WARNING: no policy specified for kudu/127.11.102.2@KRBTEST.COM; defaulting to no policy
Jun 20 06:07:03 dist-test-slave-9hm5 krb5kdc[15195](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935623, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.102.1@KRBTEST.COM for kudu/127.11.102.62@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.102.2@KRBTEST.COM" created.
I20260620 06:07:03.727803 15279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:07:03.714883 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.102.1:39815 (local address 127.11.102.62:43573)
0620 06:07:03.715019 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0620 06:07:03.715024 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 06:07:03.716033 (+ 1009us) server_negotiation.cc:407] Connection header received
0620 06:07:03.717798 (+ 1765us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:07:03.717804 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:07:03.717862 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:07:03.717960 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:07:03.719059 (+ 1099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:07:03.720009 (+ 950us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:07:03.721112 (+ 1103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:07:03.721515 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:07:03.721629 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:07:03.724073 (+ 2444us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:07:03.724076 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:07:03.724078 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:07:03.724103 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:07:03.725488 (+ 1385us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:07:03.726084 (+ 596us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:07:03.726086 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:07:03.726088 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:07:03.726198 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:07:03.726525 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:07:03.726530 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:07:03.726535 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:07:03.726809 (+ 274us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:07:03.726914 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0620 06:07:03.727469 (+ 555us) server_negotiation.cc:299] Negotiation successful
0620 06:07:03.727594 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260620 06:07:03.728565 15433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:07:03.715349 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.102.62:43573 (local address 127.11.102.1:39815)
0620 06:07:03.715860 (+ 511us) negotiation.cc:107] Waiting for socket to connect
0620 06:07:03.715918 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0620 06:07:03.717590 (+ 1672us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:07:03.718120 (+ 530us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:07:03.718127 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:07:03.718457 (+ 330us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:07:03.718910 (+ 453us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:07:03.718922 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:07:03.720140 (+ 1218us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:07:03.720143 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:07:03.720987 (+ 844us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:07:03.720995 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:07:03.721662 (+ 667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:07:03.721665 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:07:03.721801 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:07:03.722407 (+ 606us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:07:03.722422 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:07:03.723921 (+ 1499us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:07:03.725588 (+ 1667us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:07:03.725592 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:07:03.725603 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:07:03.725961 (+ 358us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:07:03.726298 (+ 337us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:07:03.726300 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:07:03.726301 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:07:03.726429 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:07:03.726923 (+ 494us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:07:03.726931 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:07:03.727212 (+ 281us) client_negotiation.cc:769] Sending connection context
0620 06:07:03.727427 (+ 215us) client_negotiation.cc:240] Negotiation successful
0620 06:07:03.727764 (+ 337us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":299,"thread_start_us":110,"threads_started":1}
I20260620 06:07:03.729796 15431 heartbeater.cc:347] Connected to a master server at 127.11.102.62:43573
I20260620 06:07:03.730059 15431 heartbeater.cc:464] Registering TS with master...
I20260620 06:07:03.730672 15431 heartbeater.cc:511] Master 127.11.102.62:43573 requested a full tablet report, sending...
I20260620 06:07:03.732429 15240 ts_manager.cc:194] Registered new tserver with Master: 9198dc22c7654a29946a3c9b9d09fcff (127.11.102.1:45403)
I20260620 06:07:03.734038 15240 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.102.1@KRBTEST.COM'} at 127.11.102.1:39815
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.102.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781935603120653-11672-0/krb5kdc/kudu_127.11.102.2.keytab.
Entry for principal kudu/127.11.102.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781935603120653-11672-0/krb5kdc/kudu_127.11.102.2.keytab.
I20260620 06:07:03.740249 11672 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9HMp43/build/debug/bin/kudu
/tmp/dist-test-task9HMp43/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781935603120653-11672-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781935603120653-11672-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781935603120653-11672-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781935603120653-11672-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.102.2:0
--local_ip_for_outbound_sockets=127.11.102.2
--webserver_interface=127.11.102.2
--webserver_port=0
--tserver_master_addrs=127.11.102.62:43573
--keytab_file=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781935603120653-11672-0/krb5kdc/kudu_127.11.102.2.keytab
--principal=kudu/127.11.102.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.102.20:35239
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781935603120653-11672-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781935603120653-11672-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781935603120653-11672-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 06:07:03.845769 15436 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:07:03.846035 15436 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:07:03.846105 15436 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:07:03.850098 15436 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:07:03.850176 15436 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:07:03.850261 15436 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.102.2
I20260620 06:07:03.855396 15436 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.102.20:35239
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781935603120653-11672-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781935603120653-11672-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781935603120653-11672-0/krb5kdc/kudu_127.11.102.2.keytab
--principal=kudu/127.11.102.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.102.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781935603120653-11672-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.11.102.2
--webserver_port=0
--tserver_master_addrs=127.11.102.62:43573
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15436
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.102.2
--log_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781935603120653-11672-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 05:28:48 UTC on ccycloud.quasar-fixhms.root.comops.site
build id 4457020
I20260620 06:07:03.856403 15436 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:07:03.857386 15436 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 06:07:03.866362 15441 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:07:03.866376 15444 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:07:03.866395 15442 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:07:03.866878 15436 server_base.cc:1034] running on GCE node
I20260620 06:07:03.867291 15436 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:07:03.867959 15436 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:07:03.869135 15436 hybrid_clock.cc:648] HybridClock initialized: now 1781935623869126 us; error 35 us; skew 500 ppm
Jun 20 06:07:03 dist-test-slave-9hm5 krb5kdc[15195](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935623, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.102.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 06:07:03.871822 15436 init.cc:377] Logged in from keytab as kudu/127.11.102.2@KRBTEST.COM (short username kudu)
I20260620 06:07:03.872783 15436 webserver.cc:460] Webserver started at http://127.11.102.2:40635/ using document root <none> and password file <none>
I20260620 06:07:03.873342 15436 fs_manager.cc:359] Metadata directory not provided
I20260620 06:07:03.873420 15436 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:07:03.873693 15436 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 06:07:03.875815 15436 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781935603120653-11672-0/minicluster-data/ts-1/data/instance:
uuid: "367238c968954659adcf47ed87a131c8"
format_stamp: "Formatted at 2026-06-20 06:07:03 on dist-test-slave-9hm5"
server_key: "74e4012f63a75c0a976a41deffe22b42"
server_key_iv: "e101ea1850169bdfb06c038878f3bd48"
server_key_version: "encryptionkey@0"
I20260620 06:07:03.876348 15436 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781935603120653-11672-0/minicluster-data/ts-1/wal/instance:
uuid: "367238c968954659adcf47ed87a131c8"
format_stamp: "Formatted at 2026-06-20 06:07:03 on dist-test-slave-9hm5"
server_key: "74e4012f63a75c0a976a41deffe22b42"
server_key_iv: "e101ea1850169bdfb06c038878f3bd48"
server_key_version: "encryptionkey@0"
I20260620 06:07:03.879813 15436 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 06:07:03.882064 15451 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:07:03.883261 15436 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 06:07:03.883386 15436 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781935603120653-11672-0/minicluster-data/ts-1/data,/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781935603120653-11672-0/minicluster-data/ts-1/wal
uuid: "367238c968954659adcf47ed87a131c8"
format_stamp: "Formatted at 2026-06-20 06:07:03 on dist-test-slave-9hm5"
server_key: "74e4012f63a75c0a976a41deffe22b42"
server_key_iv: "e101ea1850169bdfb06c038878f3bd48"
server_key_version: "encryptionkey@0"
I20260620 06:07:03.883493 15436 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781935603120653-11672-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781935603120653-11672-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781935603120653-11672-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:07:03.918548 15436 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:07:03.921693 15436 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:07:03.921897 15436 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:07:03.922474 15436 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 06:07:03.923291 15436 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 06:07:03.923363 15436 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 06:07:03.923431 15436 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 06:07:03.923484 15436 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 06:07:03.932484 15436 rpc_server.cc:307] RPC server started. Bound to: 127.11.102.2:37819
I20260620 06:07:03.932510 15564 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.102.2:37819 every 8 connection(s)
I20260620 06:07:03.933352 15436 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781935603120653-11672-0/minicluster-data/ts-1/data/info.pb
I20260620 06:07:03.935879 11672 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9HMp43/build/debug/bin/kudu as pid 15436
I20260620 06:07:03.936002 11672 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781935603120653-11672-0/minicluster-data/ts-1/wal/instance
I20260620 06:07:03.936292 11672 external_mini_cluster.cc:1428] Setting key 5ece2b05498d7620bd406bf4d5c80168
Jun 20 06:07:03 dist-test-slave-9hm5 krb5kdc[15195](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935623, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.102.2@KRBTEST.COM for kudu/127.11.102.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.11.102.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.102.3@KRBTEST.COM" created.
I20260620 06:07:03.947954 15279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:07:03.934970 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.102.2:38243 (local address 127.11.102.62:43573)
0620 06:07:03.935123 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0620 06:07:03.935127 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 06:07:03.936017 (+ 890us) server_negotiation.cc:407] Connection header received
0620 06:07:03.937581 (+ 1564us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:07:03.937584 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:07:03.937634 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:07:03.937732 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:07:03.939267 (+ 1535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:07:03.940155 (+ 888us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:07:03.941250 (+ 1095us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:07:03.941679 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:07:03.941805 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:07:03.944451 (+ 2646us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:07:03.944454 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:07:03.944456 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:07:03.944478 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:07:03.945950 (+ 1472us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:07:03.946395 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:07:03.946398 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:07:03.946399 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:07:03.946477 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:07:03.946884 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:07:03.946887 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:07:03.946888 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:07:03.947057 (+ 169us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:07:03.947133 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0620 06:07:03.947690 (+ 557us) server_negotiation.cc:299] Negotiation successful
0620 06:07:03.947784 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260620 06:07:03.948768 15567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:07:03.935358 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.102.62:43573 (local address 127.11.102.2:38243)
0620 06:07:03.935868 (+ 510us) negotiation.cc:107] Waiting for socket to connect
0620 06:07:03.935909 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0620 06:07:03.937353 (+ 1444us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:07:03.937913 (+ 560us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:07:03.937922 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:07:03.938437 (+ 515us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:07:03.939125 (+ 688us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:07:03.939146 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:07:03.940277 (+ 1131us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:07:03.940280 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:07:03.941123 (+ 843us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:07:03.941131 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:07:03.941817 (+ 686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:07:03.941820 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:07:03.941951 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:07:03.942523 (+ 572us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:07:03.942540 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:07:03.944288 (+ 1748us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:07:03.946054 (+ 1766us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:07:03.946058 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:07:03.946069 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:07:03.946297 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:07:03.946655 (+ 358us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:07:03.946658 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:07:03.946659 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:07:03.946789 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:07:03.947146 (+ 357us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:07:03.947151 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:07:03.947383 (+ 232us) client_negotiation.cc:769] Sending connection context
0620 06:07:03.947582 (+ 199us) client_negotiation.cc:240] Negotiation successful
0620 06:07:03.947949 (+ 367us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":308,"thread_start_us":129,"threads_started":1}
I20260620 06:07:03.949987 15565 heartbeater.cc:347] Connected to a master server at 127.11.102.62:43573
I20260620 06:07:03.950207 15565 heartbeater.cc:464] Registering TS with master...
I20260620 06:07:03.950685 15565 heartbeater.cc:511] Master 127.11.102.62:43573 requested a full tablet report, sending...
I20260620 06:07:03.951779 15240 ts_manager.cc:194] Registered new tserver with Master: 367238c968954659adcf47ed87a131c8 (127.11.102.2:37819)
I20260620 06:07:03.952328 15240 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.102.2@KRBTEST.COM'} at 127.11.102.2:38243
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.102.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781935603120653-11672-0/krb5kdc/kudu_127.11.102.3.keytab.
Entry for principal kudu/127.11.102.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781935603120653-11672-0/krb5kdc/kudu_127.11.102.3.keytab.
I20260620 06:07:03.963065 11672 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9HMp43/build/debug/bin/kudu
/tmp/dist-test-task9HMp43/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781935603120653-11672-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781935603120653-11672-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781935603120653-11672-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781935603120653-11672-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.102.3:0
--local_ip_for_outbound_sockets=127.11.102.3
--webserver_interface=127.11.102.3
--webserver_port=0
--tserver_master_addrs=127.11.102.62:43573
--keytab_file=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781935603120653-11672-0/krb5kdc/kudu_127.11.102.3.keytab
--principal=kudu/127.11.102.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.102.20:35239
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781935603120653-11672-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781935603120653-11672-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781935603120653-11672-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 06:07:04.069411 15570 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:07:04.069665 15570 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:07:04.069734 15570 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:07:04.073731 15570 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:07:04.073807 15570 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:07:04.073891 15570 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.102.3
I20260620 06:07:04.078941 15570 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.102.20:35239
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781935603120653-11672-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781935603120653-11672-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781935603120653-11672-0/krb5kdc/kudu_127.11.102.3.keytab
--principal=kudu/127.11.102.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.102.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781935603120653-11672-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.11.102.3
--webserver_port=0
--tserver_master_addrs=127.11.102.62:43573
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15570
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.102.3
--log_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781935603120653-11672-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 05:28:48 UTC on ccycloud.quasar-fixhms.root.comops.site
build id 4457020
I20260620 06:07:04.079975 15570 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:07:04.080930 15570 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 06:07:04.090088 15575 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:07:04.090157 15576 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:07:04.090152 15578 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:07:04.090436 15570 server_base.cc:1034] running on GCE node
I20260620 06:07:04.090858 15570 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:07:04.091456 15570 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:07:04.092662 15570 hybrid_clock.cc:648] HybridClock initialized: now 1781935624092659 us; error 45 us; skew 500 ppm
Jun 20 06:07:04 dist-test-slave-9hm5 krb5kdc[15195](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935624, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.102.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 06:07:04.095324 15570 init.cc:377] Logged in from keytab as kudu/127.11.102.3@KRBTEST.COM (short username kudu)
I20260620 06:07:04.096196 15570 webserver.cc:460] Webserver started at http://127.11.102.3:44129/ using document root <none> and password file <none>
I20260620 06:07:04.096727 15570 fs_manager.cc:359] Metadata directory not provided
I20260620 06:07:04.096814 15570 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:07:04.097050 15570 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 06:07:04.099004 15570 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781935603120653-11672-0/minicluster-data/ts-2/data/instance:
uuid: "a90a65c9f3804fee956d80b088bd767d"
format_stamp: "Formatted at 2026-06-20 06:07:04 on dist-test-slave-9hm5"
server_key: "0a3d49a210cb04f3939a0d2420ea91dc"
server_key_iv: "a3394f063bf99b586e28c1faac8d643d"
server_key_version: "encryptionkey@0"
I20260620 06:07:04.099551 15570 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781935603120653-11672-0/minicluster-data/ts-2/wal/instance:
uuid: "a90a65c9f3804fee956d80b088bd767d"
format_stamp: "Formatted at 2026-06-20 06:07:04 on dist-test-slave-9hm5"
server_key: "0a3d49a210cb04f3939a0d2420ea91dc"
server_key_iv: "a3394f063bf99b586e28c1faac8d643d"
server_key_version: "encryptionkey@0"
I20260620 06:07:04.103145 15570 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260620 06:07:04.105684 15585 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:07:04.106835 15570 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 06:07:04.106983 15570 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781935603120653-11672-0/minicluster-data/ts-2/data,/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781935603120653-11672-0/minicluster-data/ts-2/wal
uuid: "a90a65c9f3804fee956d80b088bd767d"
format_stamp: "Formatted at 2026-06-20 06:07:04 on dist-test-slave-9hm5"
server_key: "0a3d49a210cb04f3939a0d2420ea91dc"
server_key_iv: "a3394f063bf99b586e28c1faac8d643d"
server_key_version: "encryptionkey@0"
I20260620 06:07:04.107096 15570 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781935603120653-11672-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781935603120653-11672-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781935603120653-11672-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:07:04.135528 15570 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:07:04.139273 15570 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:07:04.139500 15570 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:07:04.140172 15570 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 06:07:04.141076 15570 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 06:07:04.141152 15570 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 06:07:04.141228 15570 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 06:07:04.141283 15570 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 06:07:04.150692 15570 rpc_server.cc:307] RPC server started. Bound to: 127.11.102.3:45383
I20260620 06:07:04.150715 15698 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.102.3:45383 every 8 connection(s)
I20260620 06:07:04.151701 15570 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781935603120653-11672-0/minicluster-data/ts-2/data/info.pb
I20260620 06:07:04.158922 11672 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9HMp43/build/debug/bin/kudu as pid 15570
I20260620 06:07:04.159034 11672 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781935603120653-11672-0/minicluster-data/ts-2/wal/instance
I20260620 06:07:04.159294 11672 external_mini_cluster.cc:1428] Setting key 201763883ae12ed9b9b0270e0ac0bbf6
Jun 20 06:07:04 dist-test-slave-9hm5 krb5kdc[15195](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935624, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.102.3@KRBTEST.COM for kudu/127.11.102.62@KRBTEST.COM
I20260620 06:07:04.166397 15279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:07:04.153561 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.102.3:34493 (local address 127.11.102.62:43573)
0620 06:07:04.153705 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0620 06:07:04.153709 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 06:07:04.154549 (+ 840us) server_negotiation.cc:407] Connection header received
0620 06:07:04.156256 (+ 1707us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:07:04.156260 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:07:04.156320 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:07:04.156414 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:07:04.157627 (+ 1213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:07:04.158207 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:07:04.159594 (+ 1387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:07:04.159878 (+ 284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:07:04.160004 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:07:04.162459 (+ 2455us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:07:04.162463 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:07:04.162466 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:07:04.162493 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:07:04.164144 (+ 1651us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:07:04.164685 (+ 541us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:07:04.164691 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:07:04.164693 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:07:04.164780 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:07:04.165144 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:07:04.165148 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:07:04.165150 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:07:04.165348 (+ 198us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:07:04.165463 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0620 06:07:04.166085 (+ 622us) server_negotiation.cc:299] Negotiation successful
0620 06:07:04.166206 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260620 06:07:04.167232 15701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:07:04.153923 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.102.62:43573 (local address 127.11.102.3:34493)
0620 06:07:04.154375 (+ 452us) negotiation.cc:107] Waiting for socket to connect
0620 06:07:04.154420 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0620 06:07:04.156025 (+ 1605us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:07:04.156589 (+ 564us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:07:04.156597 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:07:04.156970 (+ 373us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:07:04.157464 (+ 494us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:07:04.157477 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:07:04.158338 (+ 861us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:07:04.158343 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:07:04.159463 (+ 1120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:07:04.159473 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:07:04.160037 (+ 564us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:07:04.160041 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:07:04.160227 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:07:04.160879 (+ 652us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:07:04.160896 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:07:04.162294 (+ 1398us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:07:04.164286 (+ 1992us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:07:04.164291 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:07:04.164306 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:07:04.164574 (+ 268us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:07:04.164884 (+ 310us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:07:04.164888 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:07:04.164890 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:07:04.165033 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:07:04.165459 (+ 426us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:07:04.165465 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:07:04.165815 (+ 350us) client_negotiation.cc:769] Sending connection context
0620 06:07:04.166025 (+ 210us) client_negotiation.cc:240] Negotiation successful
0620 06:07:04.166347 (+ 322us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":261,"thread_start_us":109,"threads_started":1}
I20260620 06:07:04.168316 15699 heartbeater.cc:347] Connected to a master server at 127.11.102.62:43573
I20260620 06:07:04.168558 15699 heartbeater.cc:464] Registering TS with master...
I20260620 06:07:04.169026 15699 heartbeater.cc:511] Master 127.11.102.62:43573 requested a full tablet report, sending...
I20260620 06:07:04.169987 15240 ts_manager.cc:194] Registered new tserver with Master: a90a65c9f3804fee956d80b088bd767d (127.11.102.3:45383)
I20260620 06:07:04.170593 15240 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.102.3@KRBTEST.COM'} at 127.11.102.3:34493
I20260620 06:07:04.172977 11672 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 20 06:07:04 dist-test-slave-9hm5 krb5kdc[15195](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935624, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 20 06:07:04 dist-test-slave-9hm5 krb5kdc[15195](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935624, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.102.1@KRBTEST.COM
I20260620 06:07:04.197955 15706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:07:04.188917 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50646 (local address 127.11.102.1:45403)
0620 06:07:04.189176 (+ 259us) server_negotiation.cc:206] Beginning negotiation
0620 06:07:04.189199 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0620 06:07:04.189222 (+ 23us) server_negotiation.cc:407] Connection header received
0620 06:07:04.189331 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:07:04.189336 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:07:04.189533 (+ 197us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:07:04.189649 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:07:04.190297 (+ 648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:07:04.191073 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:07:04.191927 (+ 854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:07:04.192284 (+ 357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:07:04.192377 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:07:04.194726 (+ 2349us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:07:04.194735 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:07:04.194747 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:07:04.194772 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:07:04.196381 (+ 1609us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:07:04.196768 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:07:04.196773 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:07:04.196774 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:07:04.196849 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:07:04.197136 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:07:04.197138 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:07:04.197139 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:07:04.197367 (+ 228us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:07:04.197525 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0620 06:07:04.197689 (+ 164us) server_negotiation.cc:299] Negotiation successful
0620 06:07:04.197805 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":164,"thread_start_us":102,"threads_started":1}
Jun 20 06:07:04 dist-test-slave-9hm5 krb5kdc[15195](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935624, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.102.62@KRBTEST.COM
I20260620 06:07:04.208813 15279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:07:04.200837 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42484 (local address 127.11.102.62:43573)
0620 06:07:04.200979 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0620 06:07:04.200984 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 06:07:04.201029 (+ 45us) server_negotiation.cc:407] Connection header received
0620 06:07:04.201189 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:07:04.201192 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:07:04.201233 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:07:04.201315 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:07:04.201898 (+ 583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:07:04.202397 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:07:04.203159 (+ 762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:07:04.203375 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:07:04.203483 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:07:04.205740 (+ 2257us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:07:04.205746 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:07:04.205753 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:07:04.205775 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:07:04.207151 (+ 1376us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:07:04.207564 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:07:04.207569 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:07:04.207571 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:07:04.207699 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:07:04.207975 (+ 276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:07:04.207979 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:07:04.207980 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:07:04.208184 (+ 204us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:07:04.208325 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0620 06:07:04.208543 (+ 218us) server_negotiation.cc:299] Negotiation successful
0620 06:07:04.208655 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
W20260620 06:07:04.209194 15240 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42484
I20260620 06:07:04.209714 11672 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9HMp43/build/debug/bin/kudu with pid 15302
I20260620 06:07:04.215862 11672 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9HMp43/build/debug/bin/kudu with pid 15436
I20260620 06:07:04.220844 11672 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9HMp43/build/debug/bin/kudu with pid 15570
I20260620 06:07:04.225924 11672 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9HMp43/build/debug/bin/kudu with pid 15208
2026-06-20T06:07:04Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (2120 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781935603120653-11672-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 06:07:04 dist-test-slave-9hm5 krb5kdc[15711](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 06:07:04 dist-test-slave-9hm5 krb5kdc[15711](info): set up 2 sockets
Jun 20 06:07:04 dist-test-slave-9hm5 krb5kdc[15711](info): commencing operation
krb5kdc: starting...
W20260620 06:07:05.259779 11672 mini_kdc.cc:121] Time spent starting KDC: real 1.012s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781935603120653-11672-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781935603120653-11672-0/krb5kdc/test-user.keytab.
Jun 20 06:07:05 dist-test-slave-9hm5 krb5kdc[15711](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935625, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T06:07:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T06:07:05Z Disabled control of system clock
WARNING: no policy specified for kudu/127.11.102.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.102.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.102.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781935603120653-11672-0/krb5kdc/kudu_127.11.102.62.keytab.
Entry for principal kudu/127.11.102.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781935603120653-11672-0/krb5kdc/kudu_127.11.102.62.keytab.
I20260620 06:07:05.378662 11672 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9HMp43/build/debug/bin/kudu
/tmp/dist-test-task9HMp43/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781935603120653-11672-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781935603120653-11672-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781935603120653-11672-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781935603120653-11672-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.102.62:35515
--webserver_interface=127.11.102.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781935603120653-11672-0/krb5kdc/kudu_127.11.102.62.keytab
--principal=kudu/127.11.102.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.102.20:41759
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.102.62:35515
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781935603120653-11672-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781935603120653-11672-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781935603120653-11672-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 06:07:05.484565 15724 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:07:05.484854 15724 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:07:05.484930 15724 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:07:05.489533 15724 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 06:07:05.489611 15724 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:07:05.489645 15724 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:07:05.489682 15724 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 06:07:05.489713 15724 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 06:07:05.494889 15724 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.102.20:41759
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781935603120653-11672-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781935603120653-11672-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.102.62:35515
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781935603120653-11672-0/krb5kdc/kudu_127.11.102.62.keytab
--principal=kudu/127.11.102.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.102.62:35515
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781935603120653-11672-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.102.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15724
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781935603120653-11672-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 05:28:48 UTC on ccycloud.quasar-fixhms.root.comops.site
build id 4457020
I20260620 06:07:05.495915 15724 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:07:05.496912 15724 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 06:07:05.504763 15729 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:07:05.504777 15730 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:07:05.504825 15732 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:07:05.505123 15724 server_base.cc:1034] running on GCE node
I20260620 06:07:05.505649 15724 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:07:05.506585 15724 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:07:05.507743 15724 hybrid_clock.cc:648] HybridClock initialized: now 1781935625507726 us; error 28 us; skew 500 ppm
Jun 20 06:07:05 dist-test-slave-9hm5 krb5kdc[15711](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935625, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.102.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 06:07:05.510499 15724 init.cc:377] Logged in from keytab as kudu/127.11.102.62@KRBTEST.COM (short username kudu)
I20260620 06:07:05.511195 15724 fs_manager.cc:359] Metadata directory not provided
I20260620 06:07:05.511287 15724 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:07:05.511510 15724 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 06:07:05.513460 15724 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781935603120653-11672-0/minicluster-data/master-0/data/instance:
uuid: "1afe83e2a69e40d59ff7d0497b27392e"
format_stamp: "Formatted at 2026-06-20 06:07:05 on dist-test-slave-9hm5"
server_key: "14485f4a1e3b8295ac4f533f43cb9af6"
server_key_iv: "4ebd119745e9abe276e5bd01b014dc4c"
server_key_version: "encryptionkey@0"
I20260620 06:07:05.514078 15724 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781935603120653-11672-0/minicluster-data/master-0/wal/instance:
uuid: "1afe83e2a69e40d59ff7d0497b27392e"
format_stamp: "Formatted at 2026-06-20 06:07:05 on dist-test-slave-9hm5"
server_key: "14485f4a1e3b8295ac4f533f43cb9af6"
server_key_iv: "4ebd119745e9abe276e5bd01b014dc4c"
server_key_version: "encryptionkey@0"
I20260620 06:07:05.517594 15724 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260620 06:07:05.519862 15738 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:07:05.520934 15724 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260620 06:07:05.521071 15724 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781935603120653-11672-0/minicluster-data/master-0/data,/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781935603120653-11672-0/minicluster-data/master-0/wal
uuid: "1afe83e2a69e40d59ff7d0497b27392e"
format_stamp: "Formatted at 2026-06-20 06:07:05 on dist-test-slave-9hm5"
server_key: "14485f4a1e3b8295ac4f533f43cb9af6"
server_key_iv: "4ebd119745e9abe276e5bd01b014dc4c"
server_key_version: "encryptionkey@0"
I20260620 06:07:05.521184 15724 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781935603120653-11672-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781935603120653-11672-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781935603120653-11672-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:07:05.553306 15724 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:07:05.556684 15724 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:07:05.556902 15724 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:07:05.563995 15724 rpc_server.cc:307] RPC server started. Bound to: 127.11.102.62:35515
I20260620 06:07:05.564356 15790 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.102.62:35515 every 8 connection(s)
I20260620 06:07:05.564661 15724 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781935603120653-11672-0/minicluster-data/master-0/data/info.pb
I20260620 06:07:05.567236 15791 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 06:07:05.572438 15791 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1afe83e2a69e40d59ff7d0497b27392e: Bootstrap starting.
I20260620 06:07:05.574631 15791 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1afe83e2a69e40d59ff7d0497b27392e: Neither blocks nor log segments found. Creating new log.
I20260620 06:07:05.574862 11672 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9HMp43/build/debug/bin/kudu as pid 15724
I20260620 06:07:05.575006 11672 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781935603120653-11672-0/minicluster-data/master-0/wal/instance
I20260620 06:07:05.575253 11672 external_mini_cluster.cc:1428] Setting key 3e6275603411a8bf8665791569e1b0dc
I20260620 06:07:05.575469 15791 log.cc:824] T 00000000000000000000000000000000 P 1afe83e2a69e40d59ff7d0497b27392e: Log is configured to *not* fsync() on all Append() calls
I20260620 06:07:05.577625 15791 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1afe83e2a69e40d59ff7d0497b27392e: No bootstrap required, opened a new log
I20260620 06:07:05.581852 15791 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1afe83e2a69e40d59ff7d0497b27392e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1afe83e2a69e40d59ff7d0497b27392e" member_type: VOTER last_known_addr { host: "127.11.102.62" port: 35515 } }
I20260620 06:07:05.582103 15791 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1afe83e2a69e40d59ff7d0497b27392e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 06:07:05.582149 15791 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1afe83e2a69e40d59ff7d0497b27392e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1afe83e2a69e40d59ff7d0497b27392e, State: Initialized, Role: FOLLOWER
I20260620 06:07:05.582610 15791 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1afe83e2a69e40d59ff7d0497b27392e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1afe83e2a69e40d59ff7d0497b27392e" member_type: VOTER last_known_addr { host: "127.11.102.62" port: 35515 } }
I20260620 06:07:05.582795 15791 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1afe83e2a69e40d59ff7d0497b27392e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 06:07:05.582864 15791 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1afe83e2a69e40d59ff7d0497b27392e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 06:07:05.582970 15791 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1afe83e2a69e40d59ff7d0497b27392e [term 0 FOLLOWER]: Advancing to term 1
I20260620 06:07:05.583959 15791 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1afe83e2a69e40d59ff7d0497b27392e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1afe83e2a69e40d59ff7d0497b27392e" member_type: VOTER last_known_addr { host: "127.11.102.62" port: 35515 } }
I20260620 06:07:05.584340 15791 leader_election.cc:302] T 00000000000000000000000000000000 P 1afe83e2a69e40d59ff7d0497b27392e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1afe83e2a69e40d59ff7d0497b27392e; no voters:
Jun 20 06:07:05 dist-test-slave-9hm5 krb5kdc[15711](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935625, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.102.62@KRBTEST.COM
I20260620 06:07:05.584667 15791 leader_election.cc:288] T 00000000000000000000000000000000 P 1afe83e2a69e40d59ff7d0497b27392e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 06:07:05.584789 15796 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1afe83e2a69e40d59ff7d0497b27392e [term 1 FOLLOWER]: Leader election won for term 1
I20260620 06:07:05.585000 15796 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1afe83e2a69e40d59ff7d0497b27392e [term 1 LEADER]: Becoming Leader. State: Replica: 1afe83e2a69e40d59ff7d0497b27392e, State: Running, Role: LEADER
I20260620 06:07:05.585331 15796 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1afe83e2a69e40d59ff7d0497b27392e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1afe83e2a69e40d59ff7d0497b27392e" member_type: VOTER last_known_addr { host: "127.11.102.62" port: 35515 } }
I20260620 06:07:05.585920 15791 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1afe83e2a69e40d59ff7d0497b27392e [sys.catalog]: configured and running, proceeding with master startup.
I20260620 06:07:05.586980 15797 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1afe83e2a69e40d59ff7d0497b27392e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1afe83e2a69e40d59ff7d0497b27392e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1afe83e2a69e40d59ff7d0497b27392e" member_type: VOTER last_known_addr { host: "127.11.102.62" port: 35515 } } }
I20260620 06:07:05.587105 15797 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1afe83e2a69e40d59ff7d0497b27392e [sys.catalog]: This master's current role is: LEADER
I20260620 06:07:05.587448 15798 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1afe83e2a69e40d59ff7d0497b27392e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1afe83e2a69e40d59ff7d0497b27392e. Latest consensus state: current_term: 1 leader_uuid: "1afe83e2a69e40d59ff7d0497b27392e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1afe83e2a69e40d59ff7d0497b27392e" member_type: VOTER last_known_addr { host: "127.11.102.62" port: 35515 } } }
I20260620 06:07:05.587548 15798 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1afe83e2a69e40d59ff7d0497b27392e [sys.catalog]: This master's current role is: LEADER
I20260620 06:07:05.587879 15805 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 06:07:05.590672 15795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:07:05.576362 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39992 (local address 127.11.102.62:35515)
0620 06:07:05.576796 (+ 434us) server_negotiation.cc:206] Beginning negotiation
0620 06:07:05.576815 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0620 06:07:05.576856 (+ 41us) server_negotiation.cc:407] Connection header received
0620 06:07:05.577641 (+ 785us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:07:05.577680 (+ 39us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:07:05.578075 (+ 395us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:07:05.578421 (+ 346us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:07:05.579168 (+ 747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:07:05.580683 (+ 1515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:07:05.581554 (+ 871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:07:05.582223 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:07:05.582385 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:07:05.585197 (+ 2812us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:07:05.585220 (+ 23us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:07:05.585236 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:07:05.585266 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:07:05.587394 (+ 2128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:07:05.587948 (+ 554us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:07:05.587954 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:07:05.587960 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:07:05.588085 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:07:05.588392 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:07:05.588395 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:07:05.588397 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:07:05.588855 (+ 458us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:07:05.589053 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0620 06:07:05.589511 (+ 458us) server_negotiation.cc:299] Negotiation successful
0620 06:07:05.589783 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":290,"thread_start_us":135,"threads_started":1}
I20260620 06:07:05.590932 15805 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 06:07:05.595690 15805 catalog_manager.cc:1347] Generated new cluster ID: 490fbda562034ecbbcf253c4d9119164
I20260620 06:07:05.595752 15805 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 06:07:05.610153 15805 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 06:07:05.611007 15805 catalog_manager.cc:1504] Loading token signing keys...
I20260620 06:07:05.621089 15805 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1afe83e2a69e40d59ff7d0497b27392e: Generated new TSK 0
I20260620 06:07:05.621764 15805 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.11.102.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.102.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.102.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781935603120653-11672-0/krb5kdc/kudu_127.11.102.1.keytab.
Entry for principal kudu/127.11.102.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781935603120653-11672-0/krb5kdc/kudu_127.11.102.1.keytab.
I20260620 06:07:05.650540 11672 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9HMp43/build/debug/bin/kudu
/tmp/dist-test-task9HMp43/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781935603120653-11672-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781935603120653-11672-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781935603120653-11672-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781935603120653-11672-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.102.1:0
--local_ip_for_outbound_sockets=127.11.102.1
--webserver_interface=127.11.102.1
--webserver_port=0
--tserver_master_addrs=127.11.102.62:35515
--keytab_file=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781935603120653-11672-0/krb5kdc/kudu_127.11.102.1.keytab
--principal=kudu/127.11.102.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.102.20:41759
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781935603120653-11672-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781935603120653-11672-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781935603120653-11672-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 06:07:05.759717 15817 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:07:05.759959 15817 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:07:05.760002 15817 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:07:05.764015 15817 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:07:05.764063 15817 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:07:05.764153 15817 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.102.1
I20260620 06:07:05.769294 15817 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.102.20:41759
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781935603120653-11672-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781935603120653-11672-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781935603120653-11672-0/krb5kdc/kudu_127.11.102.1.keytab
--principal=kudu/127.11.102.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.102.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781935603120653-11672-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.11.102.1
--webserver_port=0
--tserver_master_addrs=127.11.102.62:35515
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15817
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.102.1
--log_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781935603120653-11672-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 05:28:48 UTC on ccycloud.quasar-fixhms.root.comops.site
build id 4457020
I20260620 06:07:05.770251 15817 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:07:05.771231 15817 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 06:07:05.779790 15825 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:07:05.779815 15822 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:07:05.779829 15823 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:07:05.779874 15817 server_base.cc:1034] running on GCE node
I20260620 06:07:05.780395 15817 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:07:05.780966 15817 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:07:05.782148 15817 hybrid_clock.cc:648] HybridClock initialized: now 1781935625782133 us; error 32 us; skew 500 ppm
Jun 20 06:07:05 dist-test-slave-9hm5 krb5kdc[15711](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935625, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.102.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 06:07:05.784956 15817 init.cc:377] Logged in from keytab as kudu/127.11.102.1@KRBTEST.COM (short username kudu)
I20260620 06:07:05.785647 15817 fs_manager.cc:359] Metadata directory not provided
I20260620 06:07:05.785704 15817 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:07:05.785905 15817 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 06:07:05.788004 15817 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781935603120653-11672-0/minicluster-data/ts-0/data/instance:
uuid: "51a86ef5aaa04847aafbda5c22a02594"
format_stamp: "Formatted at 2026-06-20 06:07:05 on dist-test-slave-9hm5"
server_key: "e14eec01c65a6e8a60c1f64b9c97ac97"
server_key_iv: "ff155aea8ce74f7f253871f422d34b05"
server_key_version: "encryptionkey@0"
I20260620 06:07:05.788519 15817 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781935603120653-11672-0/minicluster-data/ts-0/wal/instance:
uuid: "51a86ef5aaa04847aafbda5c22a02594"
format_stamp: "Formatted at 2026-06-20 06:07:05 on dist-test-slave-9hm5"
server_key: "e14eec01c65a6e8a60c1f64b9c97ac97"
server_key_iv: "ff155aea8ce74f7f253871f422d34b05"
server_key_version: "encryptionkey@0"
I20260620 06:07:05.791955 15817 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 06:07:05.794144 15831 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:07:05.795267 15817 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 06:07:05.795392 15817 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781935603120653-11672-0/minicluster-data/ts-0/data,/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781935603120653-11672-0/minicluster-data/ts-0/wal
uuid: "51a86ef5aaa04847aafbda5c22a02594"
format_stamp: "Formatted at 2026-06-20 06:07:05 on dist-test-slave-9hm5"
server_key: "e14eec01c65a6e8a60c1f64b9c97ac97"
server_key_iv: "ff155aea8ce74f7f253871f422d34b05"
server_key_version: "encryptionkey@0"
I20260620 06:07:05.795487 15817 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781935603120653-11672-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781935603120653-11672-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781935603120653-11672-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:07:05.817803 15817 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:07:05.821031 15817 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:07:05.821228 15817 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:07:05.821787 15817 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 06:07:05.822602 15817 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 06:07:05.822672 15817 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 06:07:05.822743 15817 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 06:07:05.822788 15817 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 06:07:05.831727 15817 rpc_server.cc:307] RPC server started. Bound to: 127.11.102.1:46757
I20260620 06:07:05.831753 15944 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.102.1:46757 every 8 connection(s)
I20260620 06:07:05.832321 15817 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781935603120653-11672-0/minicluster-data/ts-0/data/info.pb
I20260620 06:07:05.837395 11672 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9HMp43/build/debug/bin/kudu as pid 15817
I20260620 06:07:05.837535 11672 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781935603120653-11672-0/minicluster-data/ts-0/wal/instance
I20260620 06:07:05.837824 11672 external_mini_cluster.cc:1428] Setting key cb64c62bec7044a04aebdc61b6bd86bd
Jun 20 06:07:05 dist-test-slave-9hm5 krb5kdc[15711](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935625, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.102.1@KRBTEST.COM for kudu/127.11.102.62@KRBTEST.COM
I20260620 06:07:05.846778 15795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:07:05.834158 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.102.1:55515 (local address 127.11.102.62:35515)
0620 06:07:05.834321 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0620 06:07:05.834325 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 06:07:05.835496 (+ 1171us) server_negotiation.cc:407] Connection header received
0620 06:07:05.836480 (+ 984us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:07:05.836483 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:07:05.836562 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:07:05.836661 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:07:05.837913 (+ 1252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:07:05.838444 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:07:05.839760 (+ 1316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:07:05.840177 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:07:05.840301 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:07:05.842772 (+ 2471us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:07:05.842779 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:07:05.842785 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:07:05.842818 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:07:05.844633 (+ 1815us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:07:05.845141 (+ 508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:07:05.845146 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:07:05.845149 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:07:05.845248 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:07:05.845586 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:07:05.845592 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:07:05.845597 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:07:05.845835 (+ 238us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:07:05.845945 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0620 06:07:05.846419 (+ 474us) server_negotiation.cc:299] Negotiation successful
0620 06:07:05.846575 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260620 06:07:05.847541 15947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:07:05.834533 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.102.62:35515 (local address 127.11.102.1:55515)
0620 06:07:05.835337 (+ 804us) negotiation.cc:107] Waiting for socket to connect
0620 06:07:05.835377 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0620 06:07:05.836265 (+ 888us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:07:05.836836 (+ 571us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:07:05.836842 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:07:05.837225 (+ 383us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:07:05.837771 (+ 546us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:07:05.837786 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:07:05.838598 (+ 812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:07:05.838601 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:07:05.839588 (+ 987us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:07:05.839598 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:07:05.840322 (+ 724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:07:05.840325 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:07:05.840463 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:07:05.841096 (+ 633us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:07:05.841112 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:07:05.842591 (+ 1479us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:07:05.844761 (+ 2170us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:07:05.844766 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:07:05.844778 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:07:05.845029 (+ 251us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:07:05.845351 (+ 322us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:07:05.845353 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:07:05.845355 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:07:05.845487 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:07:05.845941 (+ 454us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:07:05.845946 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:07:05.846178 (+ 232us) client_negotiation.cc:769] Sending connection context
0620 06:07:05.846338 (+ 160us) client_negotiation.cc:240] Negotiation successful
0620 06:07:05.846599 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":275,"thread_start_us":110,"threads_started":1}
WARNING: no policy specified for kudu/127.11.102.2@KRBTEST.COM; defaulting to no policy
I20260620 06:07:05.848924 15945 heartbeater.cc:347] Connected to a master server at 127.11.102.62:35515
I20260620 06:07:05.849313 15945 heartbeater.cc:464] Registering TS with master...
I20260620 06:07:05.849781 15945 heartbeater.cc:511] Master 127.11.102.62:35515 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.102.2@KRBTEST.COM" created.
I20260620 06:07:05.851464 15755 ts_manager.cc:194] Registered new tserver with Master: 51a86ef5aaa04847aafbda5c22a02594 (127.11.102.1:46757)
I20260620 06:07:05.852663 15755 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.102.1@KRBTEST.COM'} at 127.11.102.1:55515
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.102.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781935603120653-11672-0/krb5kdc/kudu_127.11.102.2.keytab.
Entry for principal kudu/127.11.102.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781935603120653-11672-0/krb5kdc/kudu_127.11.102.2.keytab.
I20260620 06:07:05.864710 11672 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9HMp43/build/debug/bin/kudu
/tmp/dist-test-task9HMp43/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781935603120653-11672-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781935603120653-11672-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781935603120653-11672-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781935603120653-11672-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.102.2:0
--local_ip_for_outbound_sockets=127.11.102.2
--webserver_interface=127.11.102.2
--webserver_port=0
--tserver_master_addrs=127.11.102.62:35515
--keytab_file=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781935603120653-11672-0/krb5kdc/kudu_127.11.102.2.keytab
--principal=kudu/127.11.102.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.102.20:41759
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781935603120653-11672-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781935603120653-11672-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781935603120653-11672-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 06:07:05.971459 15950 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:07:05.971771 15950 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:07:05.971820 15950 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:07:05.975716 15950 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:07:05.975764 15950 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:07:05.975821 15950 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.102.2
I20260620 06:07:05.980954 15950 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.102.20:41759
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781935603120653-11672-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781935603120653-11672-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781935603120653-11672-0/krb5kdc/kudu_127.11.102.2.keytab
--principal=kudu/127.11.102.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.102.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781935603120653-11672-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.11.102.2
--webserver_port=0
--tserver_master_addrs=127.11.102.62:35515
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15950
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.102.2
--log_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781935603120653-11672-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 05:28:48 UTC on ccycloud.quasar-fixhms.root.comops.site
build id 4457020
I20260620 06:07:05.981880 15950 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:07:05.982801 15950 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 06:07:05.991922 15956 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:07:05.991940 15958 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:07:05.991922 15955 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:07:05.992187 15950 server_base.cc:1034] running on GCE node
I20260620 06:07:05.992583 15950 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:07:05.993206 15950 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:07:05.994398 15950 hybrid_clock.cc:648] HybridClock initialized: now 1781935625994376 us; error 44 us; skew 500 ppm
Jun 20 06:07:05 dist-test-slave-9hm5 krb5kdc[15711](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935625, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.102.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 06:07:05.997093 15950 init.cc:377] Logged in from keytab as kudu/127.11.102.2@KRBTEST.COM (short username kudu)
I20260620 06:07:05.997787 15950 fs_manager.cc:359] Metadata directory not provided
I20260620 06:07:05.997848 15950 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:07:05.998013 15950 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 06:07:06.000154 15950 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781935603120653-11672-0/minicluster-data/ts-1/data/instance:
uuid: "b4bfdf3553314427a463f1a1d17da0d3"
format_stamp: "Formatted at 2026-06-20 06:07:05 on dist-test-slave-9hm5"
server_key: "8fb41affe48f475232e284b4b22af0f0"
server_key_iv: "f2501a40d574ab229ee4cf7e9e729879"
server_key_version: "encryptionkey@0"
I20260620 06:07:06.000694 15950 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781935603120653-11672-0/minicluster-data/ts-1/wal/instance:
uuid: "b4bfdf3553314427a463f1a1d17da0d3"
format_stamp: "Formatted at 2026-06-20 06:07:05 on dist-test-slave-9hm5"
server_key: "8fb41affe48f475232e284b4b22af0f0"
server_key_iv: "f2501a40d574ab229ee4cf7e9e729879"
server_key_version: "encryptionkey@0"
I20260620 06:07:06.004303 15950 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 06:07:06.006616 15964 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:07:06.007849 15950 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 06:07:06.007972 15950 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781935603120653-11672-0/minicluster-data/ts-1/data,/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781935603120653-11672-0/minicluster-data/ts-1/wal
uuid: "b4bfdf3553314427a463f1a1d17da0d3"
format_stamp: "Formatted at 2026-06-20 06:07:05 on dist-test-slave-9hm5"
server_key: "8fb41affe48f475232e284b4b22af0f0"
server_key_iv: "f2501a40d574ab229ee4cf7e9e729879"
server_key_version: "encryptionkey@0"
I20260620 06:07:06.008069 15950 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781935603120653-11672-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781935603120653-11672-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781935603120653-11672-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:07:06.023893 15950 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:07:06.026793 15950 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:07:06.026985 15950 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:07:06.027506 15950 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 06:07:06.028290 15950 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 06:07:06.028334 15950 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 06:07:06.028400 15950 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 06:07:06.028445 15950 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 06:07:06.036942 15950 rpc_server.cc:307] RPC server started. Bound to: 127.11.102.2:45705
I20260620 06:07:06.037012 16077 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.102.2:45705 every 8 connection(s)
I20260620 06:07:06.037537 15950 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781935603120653-11672-0/minicluster-data/ts-1/data/info.pb
I20260620 06:07:06.040591 11672 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9HMp43/build/debug/bin/kudu as pid 15950
I20260620 06:07:06.040727 11672 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781935603120653-11672-0/minicluster-data/ts-1/wal/instance
I20260620 06:07:06.041038 11672 external_mini_cluster.cc:1428] Setting key a59e30d5cea56d7818c8ae9e9800dada
Jun 20 06:07:06 dist-test-slave-9hm5 krb5kdc[15711](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935625, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.102.2@KRBTEST.COM for kudu/127.11.102.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.11.102.3@KRBTEST.COM; defaulting to no policy
I20260620 06:07:06.052372 15795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:07:06.039214 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.102.2:52899 (local address 127.11.102.62:35515)
0620 06:07:06.039358 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0620 06:07:06.039364 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 06:07:06.040705 (+ 1341us) server_negotiation.cc:407] Connection header received
0620 06:07:06.041789 (+ 1084us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:07:06.041793 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:07:06.041840 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:07:06.041928 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:07:06.043430 (+ 1502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:07:06.044076 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:07:06.045180 (+ 1104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:07:06.045587 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:07:06.045710 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:07:06.048420 (+ 2710us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:07:06.048424 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:07:06.048427 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:07:06.048461 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:07:06.050190 (+ 1729us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:07:06.050709 (+ 519us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:07:06.050713 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:07:06.050715 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:07:06.050799 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:07:06.051170 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:07:06.051175 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:07:06.051177 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:07:06.051356 (+ 179us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:07:06.051465 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0620 06:07:06.052081 (+ 616us) server_negotiation.cc:299] Negotiation successful
0620 06:07:06.052207 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260620 06:07:06.053232 16080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:07:06.039667 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.102.62:35515 (local address 127.11.102.2:52899)
0620 06:07:06.040510 (+ 843us) negotiation.cc:107] Waiting for socket to connect
0620 06:07:06.040568 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0620 06:07:06.041570 (+ 1002us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:07:06.042080 (+ 510us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:07:06.042087 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:07:06.042575 (+ 488us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:07:06.043258 (+ 683us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:07:06.043280 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:07:06.044204 (+ 924us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:07:06.044207 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:07:06.045066 (+ 859us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:07:06.045074 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:07:06.045730 (+ 656us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:07:06.045733 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:07:06.045867 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:07:06.046517 (+ 650us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:07:06.046541 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:07:06.048251 (+ 1710us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:07:06.050321 (+ 2070us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:07:06.050326 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:07:06.050337 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:07:06.050598 (+ 261us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:07:06.050890 (+ 292us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:07:06.050892 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:07:06.050893 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:07:06.051060 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:07:06.051461 (+ 401us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:07:06.051467 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:07:06.051798 (+ 331us) client_negotiation.cc:769] Sending connection context
0620 06:07:06.052004 (+ 206us) client_negotiation.cc:240] Negotiation successful
0620 06:07:06.052310 (+ 306us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":258,"thread_start_us":112,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.102.3@KRBTEST.COM" created.
I20260620 06:07:06.054422 16078 heartbeater.cc:347] Connected to a master server at 127.11.102.62:35515
I20260620 06:07:06.054648 16078 heartbeater.cc:464] Registering TS with master...
I20260620 06:07:06.055199 16078 heartbeater.cc:511] Master 127.11.102.62:35515 requested a full tablet report, sending...
I20260620 06:07:06.056326 15755 ts_manager.cc:194] Registered new tserver with Master: b4bfdf3553314427a463f1a1d17da0d3 (127.11.102.2:45705)
I20260620 06:07:06.056892 15755 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.102.2@KRBTEST.COM'} at 127.11.102.2:52899
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.102.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781935603120653-11672-0/krb5kdc/kudu_127.11.102.3.keytab.
Entry for principal kudu/127.11.102.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781935603120653-11672-0/krb5kdc/kudu_127.11.102.3.keytab.
I20260620 06:07:06.068850 11672 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9HMp43/build/debug/bin/kudu
/tmp/dist-test-task9HMp43/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781935603120653-11672-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781935603120653-11672-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781935603120653-11672-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781935603120653-11672-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.102.3:0
--local_ip_for_outbound_sockets=127.11.102.3
--webserver_interface=127.11.102.3
--webserver_port=0
--tserver_master_addrs=127.11.102.62:35515
--keytab_file=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781935603120653-11672-0/krb5kdc/kudu_127.11.102.3.keytab
--principal=kudu/127.11.102.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.102.20:41759
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781935603120653-11672-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781935603120653-11672-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781935603120653-11672-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 06:07:06.175653 16083 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:07:06.175902 16083 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:07:06.175945 16083 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:07:06.180106 16083 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:07:06.180162 16083 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:07:06.180248 16083 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.102.3
I20260620 06:07:06.185391 16083 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.102.20:41759
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781935603120653-11672-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781935603120653-11672-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781935603120653-11672-0/krb5kdc/kudu_127.11.102.3.keytab
--principal=kudu/127.11.102.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.102.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781935603120653-11672-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.11.102.3
--webserver_port=0
--tserver_master_addrs=127.11.102.62:35515
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16083
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.102.3
--log_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781935603120653-11672-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 05:28:48 UTC on ccycloud.quasar-fixhms.root.comops.site
build id 4457020
I20260620 06:07:06.186456 16083 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:07:06.187455 16083 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 06:07:06.196599 16091 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:07:06.196616 16088 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:07:06.196900 16089 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:07:06.197008 16083 server_base.cc:1034] running on GCE node
I20260620 06:07:06.197402 16083 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:07:06.197968 16083 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:07:06.199182 16083 hybrid_clock.cc:648] HybridClock initialized: now 1781935626199172 us; error 30 us; skew 500 ppm
Jun 20 06:07:06 dist-test-slave-9hm5 krb5kdc[15711](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935626, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.102.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 06:07:06.201918 16083 init.cc:377] Logged in from keytab as kudu/127.11.102.3@KRBTEST.COM (short username kudu)
I20260620 06:07:06.202608 16083 fs_manager.cc:359] Metadata directory not provided
I20260620 06:07:06.202702 16083 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:07:06.202934 16083 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 06:07:06.205133 16083 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781935603120653-11672-0/minicluster-data/ts-2/data/instance:
uuid: "5d07ffe5c010499eb9fc20e6702bbd3a"
format_stamp: "Formatted at 2026-06-20 06:07:06 on dist-test-slave-9hm5"
server_key: "bce0fbf60a037adf1b9752883fc913df"
server_key_iv: "5545077009f728fb29729e38a9c903a2"
server_key_version: "encryptionkey@0"
I20260620 06:07:06.205678 16083 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781935603120653-11672-0/minicluster-data/ts-2/wal/instance:
uuid: "5d07ffe5c010499eb9fc20e6702bbd3a"
format_stamp: "Formatted at 2026-06-20 06:07:06 on dist-test-slave-9hm5"
server_key: "bce0fbf60a037adf1b9752883fc913df"
server_key_iv: "5545077009f728fb29729e38a9c903a2"
server_key_version: "encryptionkey@0"
I20260620 06:07:06.209198 16083 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260620 06:07:06.211515 16097 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:07:06.212775 16083 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 06:07:06.212920 16083 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781935603120653-11672-0/minicluster-data/ts-2/data,/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781935603120653-11672-0/minicluster-data/ts-2/wal
uuid: "5d07ffe5c010499eb9fc20e6702bbd3a"
format_stamp: "Formatted at 2026-06-20 06:07:06 on dist-test-slave-9hm5"
server_key: "bce0fbf60a037adf1b9752883fc913df"
server_key_iv: "5545077009f728fb29729e38a9c903a2"
server_key_version: "encryptionkey@0"
I20260620 06:07:06.213027 16083 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781935603120653-11672-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781935603120653-11672-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781935603120653-11672-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:07:06.229466 16083 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:07:06.232436 16083 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:07:06.232630 16083 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:07:06.233186 16083 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 06:07:06.233974 16083 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 06:07:06.234048 16083 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 06:07:06.234123 16083 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 06:07:06.234169 16083 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 06:07:06.242353 16083 rpc_server.cc:307] RPC server started. Bound to: 127.11.102.3:42949
I20260620 06:07:06.242374 16210 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.102.3:42949 every 8 connection(s)
I20260620 06:07:06.242885 16083 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781935603120653-11672-0/minicluster-data/ts-2/data/info.pb
I20260620 06:07:06.244534 11672 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9HMp43/build/debug/bin/kudu as pid 16083
I20260620 06:07:06.244650 11672 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781935603120653-11672-0/minicluster-data/ts-2/wal/instance
I20260620 06:07:06.244956 11672 external_mini_cluster.cc:1428] Setting key 96cad1dc202950f531bd78a215e339f5
Jun 20 06:07:06 dist-test-slave-9hm5 krb5kdc[15711](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935626, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.102.3@KRBTEST.COM for kudu/127.11.102.62@KRBTEST.COM
I20260620 06:07:06.255962 15795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:07:06.244492 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.102.3:35279 (local address 127.11.102.62:35515)
0620 06:07:06.244651 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0620 06:07:06.244656 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 06:07:06.246185 (+ 1529us) server_negotiation.cc:407] Connection header received
0620 06:07:06.247258 (+ 1073us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:07:06.247286 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:07:06.247341 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:07:06.247406 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:07:06.248596 (+ 1190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:07:06.249096 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:07:06.250195 (+ 1099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:07:06.250425 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:07:06.250504 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:07:06.252819 (+ 2315us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:07:06.252822 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:07:06.252824 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:07:06.252847 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:07:06.254113 (+ 1266us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:07:06.254601 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:07:06.254603 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:07:06.254604 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:07:06.254680 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:07:06.254979 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:07:06.254981 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:07:06.254982 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:07:06.255147 (+ 165us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:07:06.255223 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0620 06:07:06.255754 (+ 531us) server_negotiation.cc:299] Negotiation successful
0620 06:07:06.255848 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260620 06:07:06.256752 16213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:07:06.244984 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.102.62:35515 (local address 127.11.102.3:35279)
0620 06:07:06.246022 (+ 1038us) negotiation.cc:107] Waiting for socket to connect
0620 06:07:06.246078 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0620 06:07:06.247043 (+ 965us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:07:06.247564 (+ 521us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:07:06.247570 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:07:06.247939 (+ 369us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:07:06.248472 (+ 533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:07:06.248487 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:07:06.249221 (+ 734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:07:06.249223 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:07:06.250084 (+ 861us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:07:06.250093 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:07:06.250554 (+ 461us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:07:06.250556 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:07:06.250700 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:07:06.251225 (+ 525us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:07:06.251239 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:07:06.252676 (+ 1437us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:07:06.254226 (+ 1550us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:07:06.254231 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:07:06.254241 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:07:06.254511 (+ 270us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:07:06.254760 (+ 249us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:07:06.254762 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:07:06.254763 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:07:06.254892 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:07:06.255267 (+ 375us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:07:06.255272 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:07:06.255501 (+ 229us) client_negotiation.cc:769] Sending connection context
0620 06:07:06.255722 (+ 221us) client_negotiation.cc:240] Negotiation successful
0620 06:07:06.255970 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":342,"thread_start_us":142,"threads_started":1}
I20260620 06:07:06.257877 16211 heartbeater.cc:347] Connected to a master server at 127.11.102.62:35515
I20260620 06:07:06.258137 16211 heartbeater.cc:464] Registering TS with master...
I20260620 06:07:06.258584 16211 heartbeater.cc:511] Master 127.11.102.62:35515 requested a full tablet report, sending...
I20260620 06:07:06.259577 15755 ts_manager.cc:194] Registered new tserver with Master: 5d07ffe5c010499eb9fc20e6702bbd3a (127.11.102.3:42949)
I20260620 06:07:06.260136 15755 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.102.3@KRBTEST.COM'} at 127.11.102.3:35279
I20260620 06:07:06.270378 11672 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 06:07:06.278757 15795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:07:06.272220 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39994 (local address 127.11.102.62:35515)
0620 06:07:06.272338 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0620 06:07:06.272342 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 06:07:06.272512 (+ 170us) server_negotiation.cc:407] Connection header received
0620 06:07:06.272600 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:07:06.272603 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:07:06.272638 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:07:06.272701 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:07:06.273279 (+ 578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:07:06.273762 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:07:06.274543 (+ 781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:07:06.274739 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:07:06.274812 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:07:06.276037 (+ 1225us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:07:06.276040 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:07:06.276042 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:07:06.276062 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:07:06.277475 (+ 1413us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:07:06.277826 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:07:06.277828 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:07:06.277829 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:07:06.277900 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:07:06.278115 (+ 215us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:07:06.278117 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:07:06.278118 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:07:06.278300 (+ 182us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:07:06.278411 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0620 06:07:06.278571 (+ 160us) server_negotiation.cc:299] Negotiation successful
0620 06:07:06.278652 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260620 06:07:06.281253 15755 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39994:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 06:07:06.284142 15755 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 06:07:06.297663 16222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:07:06.292125 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.102.3:42949 (local address 127.0.0.1:36494)
0620 06:07:06.292756 (+ 631us) negotiation.cc:107] Waiting for socket to connect
0620 06:07:06.292821 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0620 06:07:06.292938 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:07:06.293786 (+ 848us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:07:06.293792 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:07:06.293820 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 06:07:06.294030 (+ 210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:07:06.294042 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:07:06.295084 (+ 1042us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:07:06.295088 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:07:06.296324 (+ 1236us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:07:06.296332 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:07:06.297149 (+ 817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:07:06.297153 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:07:06.297264 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:07:06.297302 (+ 38us) client_negotiation.cc:769] Sending connection context
0620 06:07:06.297411 (+ 109us) client_negotiation.cc:240] Negotiation successful
0620 06:07:06.297513 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":560,"mutex_wait_us":45,"spinlock_wait_cycles":3840,"thread_start_us":87,"threads_started":1}
I20260620 06:07:06.297688 16227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:07:06.292731 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36494 (local address 127.11.102.3:42949)
0620 06:07:06.293129 (+ 398us) server_negotiation.cc:206] Beginning negotiation
0620 06:07:06.293141 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 06:07:06.293165 (+ 24us) server_negotiation.cc:407] Connection header received
0620 06:07:06.293243 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:07:06.293249 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:07:06.293484 (+ 235us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:07:06.293637 (+ 153us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 06:07:06.294168 (+ 531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:07:06.294956 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:07:06.296448 (+ 1492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:07:06.297032 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:07:06.297145 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:07:06.297298 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0620 06:07:06.297446 (+ 148us) server_negotiation.cc:299] Negotiation successful
0620 06:07:06.297537 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":305,"thread_start_us":108,"threads_started":1}
I20260620 06:07:06.298555 16224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:07:06.292281 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.102.2:45705 (local address 127.0.0.1:43940)
0620 06:07:06.293291 (+ 1010us) negotiation.cc:107] Waiting for socket to connect
0620 06:07:06.293304 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 06:07:06.293401 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:07:06.293837 (+ 436us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:07:06.293840 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:07:06.293867 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 06:07:06.294037 (+ 170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:07:06.294048 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:07:06.295524 (+ 1476us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:07:06.295527 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:07:06.296936 (+ 1409us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:07:06.296947 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:07:06.298163 (+ 1216us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:07:06.298165 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:07:06.298234 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:07:06.298250 (+ 16us) client_negotiation.cc:769] Sending connection context
0620 06:07:06.298344 (+ 94us) client_negotiation.cc:240] Negotiation successful
0620 06:07:06.298413 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":958,"mutex_wait_us":129,"spinlock_wait_cycles":1664,"thread_start_us":81,"threads_started":1}
I20260620 06:07:06.298839 16225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:07:06.291931 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.102.1:46757 (local address 127.0.0.1:49082)
0620 06:07:06.292630 (+ 699us) negotiation.cc:107] Waiting for socket to connect
0620 06:07:06.293248 (+ 618us) client_negotiation.cc:174] Beginning negotiation
0620 06:07:06.293343 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:07:06.294082 (+ 739us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:07:06.294088 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:07:06.294111 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 06:07:06.294192 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:07:06.294197 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:07:06.295524 (+ 1327us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:07:06.295527 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:07:06.296944 (+ 1417us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:07:06.296951 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:07:06.298367 (+ 1416us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:07:06.298370 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:07:06.298447 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:07:06.298466 (+ 19us) client_negotiation.cc:769] Sending connection context
0620 06:07:06.298523 (+ 57us) client_negotiation.cc:240] Negotiation successful
0620 06:07:06.298585 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":529,"mutex_wait_us":154,"thread_start_us":87,"threads_started":1}
I20260620 06:07:06.298981 16223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:07:06.292049 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49082 (local address 127.11.102.1:46757)
0620 06:07:06.293637 (+ 1588us) server_negotiation.cc:206] Beginning negotiation
0620 06:07:06.293649 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 06:07:06.293669 (+ 20us) server_negotiation.cc:407] Connection header received
0620 06:07:06.293738 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:07:06.293742 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:07:06.293927 (+ 185us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:07:06.294090 (+ 163us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 06:07:06.294281 (+ 191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:07:06.295371 (+ 1090us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:07:06.297070 (+ 1699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:07:06.298237 (+ 1167us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:07:06.298371 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:07:06.298494 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0620 06:07:06.298547 (+ 53us) server_negotiation.cc:299] Negotiation successful
0620 06:07:06.298840 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1489,"thread_start_us":145,"threads_started":1}
I20260620 06:07:06.299295 16226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:07:06.292731 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43940 (local address 127.11.102.2:45705)
0620 06:07:06.293174 (+ 443us) server_negotiation.cc:206] Beginning negotiation
0620 06:07:06.293185 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 06:07:06.293407 (+ 222us) server_negotiation.cc:407] Connection header received
0620 06:07:06.293496 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:07:06.293500 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:07:06.293700 (+ 200us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:07:06.293861 (+ 161us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 06:07:06.294169 (+ 308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:07:06.295355 (+ 1186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:07:06.297107 (+ 1752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:07:06.297910 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:07:06.298007 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:07:06.298115 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0620 06:07:06.299091 (+ 976us) server_negotiation.cc:299] Negotiation successful
0620 06:07:06.299178 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":362,"thread_start_us":108,"threads_started":1}
I20260620 06:07:06.300936 16145 tablet_service.cc:1467] Processing CreateTablet for tablet dee368d1441e4c64903bbb2bc3590fc7 (DEFAULT_TABLE table=test-table [id=701212e42e554a22bf3c557a9f847696]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 06:07:06.301282 15879 tablet_service.cc:1467] Processing CreateTablet for tablet dee368d1441e4c64903bbb2bc3590fc7 (DEFAULT_TABLE table=test-table [id=701212e42e554a22bf3c557a9f847696]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 06:07:06.302091 16145 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dee368d1441e4c64903bbb2bc3590fc7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 06:07:06.302219 15879 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dee368d1441e4c64903bbb2bc3590fc7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 06:07:06.301854 16012 tablet_service.cc:1467] Processing CreateTablet for tablet dee368d1441e4c64903bbb2bc3590fc7 (DEFAULT_TABLE table=test-table [id=701212e42e554a22bf3c557a9f847696]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 06:07:06.302749 16012 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dee368d1441e4c64903bbb2bc3590fc7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 06:07:06.307715 16228 tablet_bootstrap.cc:492] T dee368d1441e4c64903bbb2bc3590fc7 P 51a86ef5aaa04847aafbda5c22a02594: Bootstrap starting.
I20260620 06:07:06.309468 16230 tablet_bootstrap.cc:492] T dee368d1441e4c64903bbb2bc3590fc7 P b4bfdf3553314427a463f1a1d17da0d3: Bootstrap starting.
I20260620 06:07:06.310026 16228 tablet_bootstrap.cc:654] T dee368d1441e4c64903bbb2bc3590fc7 P 51a86ef5aaa04847aafbda5c22a02594: Neither blocks nor log segments found. Creating new log.
I20260620 06:07:06.310000 16229 tablet_bootstrap.cc:492] T dee368d1441e4c64903bbb2bc3590fc7 P 5d07ffe5c010499eb9fc20e6702bbd3a: Bootstrap starting.
I20260620 06:07:06.311033 16228 log.cc:824] T dee368d1441e4c64903bbb2bc3590fc7 P 51a86ef5aaa04847aafbda5c22a02594: Log is configured to *not* fsync() on all Append() calls
I20260620 06:07:06.311497 16230 tablet_bootstrap.cc:654] T dee368d1441e4c64903bbb2bc3590fc7 P b4bfdf3553314427a463f1a1d17da0d3: Neither blocks nor log segments found. Creating new log.
I20260620 06:07:06.312170 16229 tablet_bootstrap.cc:654] T dee368d1441e4c64903bbb2bc3590fc7 P 5d07ffe5c010499eb9fc20e6702bbd3a: Neither blocks nor log segments found. Creating new log.
I20260620 06:07:06.312677 16230 log.cc:824] T dee368d1441e4c64903bbb2bc3590fc7 P b4bfdf3553314427a463f1a1d17da0d3: Log is configured to *not* fsync() on all Append() calls
I20260620 06:07:06.312928 16228 tablet_bootstrap.cc:492] T dee368d1441e4c64903bbb2bc3590fc7 P 51a86ef5aaa04847aafbda5c22a02594: No bootstrap required, opened a new log
I20260620 06:07:06.313122 16228 ts_tablet_manager.cc:1397] T dee368d1441e4c64903bbb2bc3590fc7 P 51a86ef5aaa04847aafbda5c22a02594: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260620 06:07:06.313472 16229 log.cc:824] T dee368d1441e4c64903bbb2bc3590fc7 P 5d07ffe5c010499eb9fc20e6702bbd3a: Log is configured to *not* fsync() on all Append() calls
I20260620 06:07:06.314673 16230 tablet_bootstrap.cc:492] T dee368d1441e4c64903bbb2bc3590fc7 P b4bfdf3553314427a463f1a1d17da0d3: No bootstrap required, opened a new log
I20260620 06:07:06.314881 16230 ts_tablet_manager.cc:1397] T dee368d1441e4c64903bbb2bc3590fc7 P b4bfdf3553314427a463f1a1d17da0d3: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.004s
I20260620 06:07:06.315395 16229 tablet_bootstrap.cc:492] T dee368d1441e4c64903bbb2bc3590fc7 P 5d07ffe5c010499eb9fc20e6702bbd3a: No bootstrap required, opened a new log
I20260620 06:07:06.315569 16229 ts_tablet_manager.cc:1397] T dee368d1441e4c64903bbb2bc3590fc7 P 5d07ffe5c010499eb9fc20e6702bbd3a: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260620 06:07:06.316905 16228 raft_consensus.cc:348] T dee368d1441e4c64903bbb2bc3590fc7 P 51a86ef5aaa04847aafbda5c22a02594 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51a86ef5aaa04847aafbda5c22a02594" member_type: VOTER last_known_addr { host: "127.11.102.1" port: 46757 } } peers { permanent_uuid: "5d07ffe5c010499eb9fc20e6702bbd3a" member_type: VOTER last_known_addr { host: "127.11.102.3" port: 42949 } } peers { permanent_uuid: "b4bfdf3553314427a463f1a1d17da0d3" member_type: VOTER last_known_addr { host: "127.11.102.2" port: 45705 } }
I20260620 06:07:06.317157 16228 raft_consensus.cc:374] T dee368d1441e4c64903bbb2bc3590fc7 P 51a86ef5aaa04847aafbda5c22a02594 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 06:07:06.317215 16228 raft_consensus.cc:729] T dee368d1441e4c64903bbb2bc3590fc7 P 51a86ef5aaa04847aafbda5c22a02594 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51a86ef5aaa04847aafbda5c22a02594, State: Initialized, Role: FOLLOWER
I20260620 06:07:06.317651 16228 consensus_queue.cc:260] T dee368d1441e4c64903bbb2bc3590fc7 P 51a86ef5aaa04847aafbda5c22a02594 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51a86ef5aaa04847aafbda5c22a02594" member_type: VOTER last_known_addr { host: "127.11.102.1" port: 46757 } } peers { permanent_uuid: "5d07ffe5c010499eb9fc20e6702bbd3a" member_type: VOTER last_known_addr { host: "127.11.102.3" port: 42949 } } peers { permanent_uuid: "b4bfdf3553314427a463f1a1d17da0d3" member_type: VOTER last_known_addr { host: "127.11.102.2" port: 45705 } }
I20260620 06:07:06.318137 16230 raft_consensus.cc:348] T dee368d1441e4c64903bbb2bc3590fc7 P b4bfdf3553314427a463f1a1d17da0d3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51a86ef5aaa04847aafbda5c22a02594" member_type: VOTER last_known_addr { host: "127.11.102.1" port: 46757 } } peers { permanent_uuid: "5d07ffe5c010499eb9fc20e6702bbd3a" member_type: VOTER last_known_addr { host: "127.11.102.3" port: 42949 } } peers { permanent_uuid: "b4bfdf3553314427a463f1a1d17da0d3" member_type: VOTER last_known_addr { host: "127.11.102.2" port: 45705 } }
I20260620 06:07:06.318388 16230 raft_consensus.cc:374] T dee368d1441e4c64903bbb2bc3590fc7 P b4bfdf3553314427a463f1a1d17da0d3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 06:07:06.318441 16230 raft_consensus.cc:729] T dee368d1441e4c64903bbb2bc3590fc7 P b4bfdf3553314427a463f1a1d17da0d3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b4bfdf3553314427a463f1a1d17da0d3, State: Initialized, Role: FOLLOWER
I20260620 06:07:06.318617 16228 ts_tablet_manager.cc:1428] T dee368d1441e4c64903bbb2bc3590fc7 P 51a86ef5aaa04847aafbda5c22a02594: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260620 06:07:06.318892 16230 consensus_queue.cc:260] T dee368d1441e4c64903bbb2bc3590fc7 P b4bfdf3553314427a463f1a1d17da0d3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51a86ef5aaa04847aafbda5c22a02594" member_type: VOTER last_known_addr { host: "127.11.102.1" port: 46757 } } peers { permanent_uuid: "5d07ffe5c010499eb9fc20e6702bbd3a" member_type: VOTER last_known_addr { host: "127.11.102.3" port: 42949 } } peers { permanent_uuid: "b4bfdf3553314427a463f1a1d17da0d3" member_type: VOTER last_known_addr { host: "127.11.102.2" port: 45705 } }
I20260620 06:07:06.318975 15945 heartbeater.cc:503] Master 127.11.102.62:35515 was elected leader, sending a full tablet report...
I20260620 06:07:06.319272 16229 raft_consensus.cc:348] T dee368d1441e4c64903bbb2bc3590fc7 P 5d07ffe5c010499eb9fc20e6702bbd3a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51a86ef5aaa04847aafbda5c22a02594" member_type: VOTER last_known_addr { host: "127.11.102.1" port: 46757 } } peers { permanent_uuid: "5d07ffe5c010499eb9fc20e6702bbd3a" member_type: VOTER last_known_addr { host: "127.11.102.3" port: 42949 } } peers { permanent_uuid: "b4bfdf3553314427a463f1a1d17da0d3" member_type: VOTER last_known_addr { host: "127.11.102.2" port: 45705 } }
I20260620 06:07:06.319501 16078 heartbeater.cc:503] Master 127.11.102.62:35515 was elected leader, sending a full tablet report...
I20260620 06:07:06.319517 16229 raft_consensus.cc:374] T dee368d1441e4c64903bbb2bc3590fc7 P 5d07ffe5c010499eb9fc20e6702bbd3a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 06:07:06.319561 16229 raft_consensus.cc:729] T dee368d1441e4c64903bbb2bc3590fc7 P 5d07ffe5c010499eb9fc20e6702bbd3a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5d07ffe5c010499eb9fc20e6702bbd3a, State: Initialized, Role: FOLLOWER
I20260620 06:07:06.319819 16230 ts_tablet_manager.cc:1428] T dee368d1441e4c64903bbb2bc3590fc7 P b4bfdf3553314427a463f1a1d17da0d3: Time spent starting tablet: real 0.005s user 0.005s sys 0.001s
I20260620 06:07:06.319986 16229 consensus_queue.cc:260] T dee368d1441e4c64903bbb2bc3590fc7 P 5d07ffe5c010499eb9fc20e6702bbd3a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51a86ef5aaa04847aafbda5c22a02594" member_type: VOTER last_known_addr { host: "127.11.102.1" port: 46757 } } peers { permanent_uuid: "5d07ffe5c010499eb9fc20e6702bbd3a" member_type: VOTER last_known_addr { host: "127.11.102.3" port: 42949 } } peers { permanent_uuid: "b4bfdf3553314427a463f1a1d17da0d3" member_type: VOTER last_known_addr { host: "127.11.102.2" port: 45705 } }
I20260620 06:07:06.320595 16211 heartbeater.cc:503] Master 127.11.102.62:35515 was elected leader, sending a full tablet report...
I20260620 06:07:06.320848 16229 ts_tablet_manager.cc:1428] T dee368d1441e4c64903bbb2bc3590fc7 P 5d07ffe5c010499eb9fc20e6702bbd3a: Time spent starting tablet: real 0.005s user 0.003s sys 0.002s
W20260620 06:07:06.333415 15946 tablet.cc:2369] T dee368d1441e4c64903bbb2bc3590fc7 P 51a86ef5aaa04847aafbda5c22a02594: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 06:07:06.493743 16212 tablet.cc:2369] T dee368d1441e4c64903bbb2bc3590fc7 P 5d07ffe5c010499eb9fc20e6702bbd3a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 06:07:06.533542 16234 raft_consensus.cc:482] T dee368d1441e4c64903bbb2bc3590fc7 P 51a86ef5aaa04847aafbda5c22a02594 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 06:07:06.533820 16234 raft_consensus.cc:504] T dee368d1441e4c64903bbb2bc3590fc7 P 51a86ef5aaa04847aafbda5c22a02594 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51a86ef5aaa04847aafbda5c22a02594" member_type: VOTER last_known_addr { host: "127.11.102.1" port: 46757 } } peers { permanent_uuid: "5d07ffe5c010499eb9fc20e6702bbd3a" member_type: VOTER last_known_addr { host: "127.11.102.3" port: 42949 } } peers { permanent_uuid: "b4bfdf3553314427a463f1a1d17da0d3" member_type: VOTER last_known_addr { host: "127.11.102.2" port: 45705 } }
I20260620 06:07:06.534960 16234 leader_election.cc:288] T dee368d1441e4c64903bbb2bc3590fc7 P 51a86ef5aaa04847aafbda5c22a02594 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5d07ffe5c010499eb9fc20e6702bbd3a (127.11.102.3:42949), b4bfdf3553314427a463f1a1d17da0d3 (127.11.102.2:45705)
W20260620 06:07:06.538523 16079 tablet.cc:2369] T dee368d1441e4c64903bbb2bc3590fc7 P b4bfdf3553314427a463f1a1d17da0d3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 06:07:06.540186 16227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:07:06.535254 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.102.1:53361 (local address 127.11.102.3:42949)
0620 06:07:06.535383 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0620 06:07:06.535388 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 06:07:06.535568 (+ 180us) server_negotiation.cc:407] Connection header received
0620 06:07:06.535745 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:07:06.535748 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:07:06.535803 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:07:06.535889 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 06:07:06.536343 (+ 454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:07:06.537117 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:07:06.538923 (+ 1806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:07:06.539513 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:07:06.539654 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:07:06.539717 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0620 06:07:06.539962 (+ 245us) server_negotiation.cc:299] Negotiation successful
0620 06:07:06.540034 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260620 06:07:06.540189 16237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:07:06.535151 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.102.3:42949 (local address 127.11.102.1:53361)
0620 06:07:06.535467 (+ 316us) negotiation.cc:107] Waiting for socket to connect
0620 06:07:06.535481 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 06:07:06.535595 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:07:06.535927 (+ 332us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:07:06.535930 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:07:06.535974 (+ 44us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 06:07:06.536223 (+ 249us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:07:06.536230 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:07:06.537227 (+ 997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:07:06.537230 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:07:06.538796 (+ 1566us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:07:06.538806 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:07:06.539695 (+ 889us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:07:06.539701 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:07:06.539812 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:07:06.539848 (+ 36us) client_negotiation.cc:769] Sending connection context
0620 06:07:06.539930 (+ 82us) client_negotiation.cc:240] Negotiation successful
0620 06:07:06.540005 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":249,"mutex_wait_us":214,"spinlock_wait_cycles":9216,"thread_start_us":98,"threads_started":1}
I20260620 06:07:06.540313 16226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:07:06.535310 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.102.1:36625 (local address 127.11.102.2:45705)
0620 06:07:06.535434 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0620 06:07:06.535440 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 06:07:06.535651 (+ 211us) server_negotiation.cc:407] Connection header received
0620 06:07:06.535783 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:07:06.535786 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:07:06.535834 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:07:06.535911 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 06:07:06.536343 (+ 432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:07:06.537117 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:07:06.539125 (+ 2008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:07:06.539783 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:07:06.539879 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:07:06.539945 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0620 06:07:06.540135 (+ 190us) server_negotiation.cc:299] Negotiation successful
0620 06:07:06.540205 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260620 06:07:06.540520 16238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:07:06.535184 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.102.2:45705 (local address 127.11.102.1:36625)
0620 06:07:06.535554 (+ 370us) negotiation.cc:107] Waiting for socket to connect
0620 06:07:06.535566 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 06:07:06.535684 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:07:06.535927 (+ 243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:07:06.535930 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:07:06.535973 (+ 43us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 06:07:06.536215 (+ 242us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:07:06.536239 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:07:06.537231 (+ 992us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:07:06.537233 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:07:06.538797 (+ 1564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:07:06.538806 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:07:06.539895 (+ 1089us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:07:06.539901 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:07:06.540007 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:07:06.540026 (+ 19us) client_negotiation.cc:769] Sending connection context
0620 06:07:06.540096 (+ 70us) client_negotiation.cc:240] Negotiation successful
0620 06:07:06.540146 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":313,"mutex_wait_us":169,"spinlock_wait_cycles":37376,"thread_start_us":117,"threads_started":1}
I20260620 06:07:06.541009 16165 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dee368d1441e4c64903bbb2bc3590fc7" candidate_uuid: "51a86ef5aaa04847aafbda5c22a02594" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5d07ffe5c010499eb9fc20e6702bbd3a" is_pre_election: true
I20260620 06:07:06.541093 16032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dee368d1441e4c64903bbb2bc3590fc7" candidate_uuid: "51a86ef5aaa04847aafbda5c22a02594" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b4bfdf3553314427a463f1a1d17da0d3" is_pre_election: true
I20260620 06:07:06.541357 16165 raft_consensus.cc:2454] T dee368d1441e4c64903bbb2bc3590fc7 P 5d07ffe5c010499eb9fc20e6702bbd3a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 51a86ef5aaa04847aafbda5c22a02594 in term 0.
I20260620 06:07:06.541407 16032 raft_consensus.cc:2454] T dee368d1441e4c64903bbb2bc3590fc7 P b4bfdf3553314427a463f1a1d17da0d3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 51a86ef5aaa04847aafbda5c22a02594 in term 0.
I20260620 06:07:06.541864 15832 leader_election.cc:302] T dee368d1441e4c64903bbb2bc3590fc7 P 51a86ef5aaa04847aafbda5c22a02594 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 51a86ef5aaa04847aafbda5c22a02594, 5d07ffe5c010499eb9fc20e6702bbd3a; no voters:
I20260620 06:07:06.542148 16234 raft_consensus.cc:2781] T dee368d1441e4c64903bbb2bc3590fc7 P 51a86ef5aaa04847aafbda5c22a02594 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 06:07:06.542232 16234 raft_consensus.cc:482] T dee368d1441e4c64903bbb2bc3590fc7 P 51a86ef5aaa04847aafbda5c22a02594 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 06:07:06.542274 16234 raft_consensus.cc:3037] T dee368d1441e4c64903bbb2bc3590fc7 P 51a86ef5aaa04847aafbda5c22a02594 [term 0 FOLLOWER]: Advancing to term 1
I20260620 06:07:06.543257 16234 raft_consensus.cc:504] T dee368d1441e4c64903bbb2bc3590fc7 P 51a86ef5aaa04847aafbda5c22a02594 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51a86ef5aaa04847aafbda5c22a02594" member_type: VOTER last_known_addr { host: "127.11.102.1" port: 46757 } } peers { permanent_uuid: "5d07ffe5c010499eb9fc20e6702bbd3a" member_type: VOTER last_known_addr { host: "127.11.102.3" port: 42949 } } peers { permanent_uuid: "b4bfdf3553314427a463f1a1d17da0d3" member_type: VOTER last_known_addr { host: "127.11.102.2" port: 45705 } }
I20260620 06:07:06.543583 16234 leader_election.cc:288] T dee368d1441e4c64903bbb2bc3590fc7 P 51a86ef5aaa04847aafbda5c22a02594 [CANDIDATE]: Term 1 election: Requested vote from peers 5d07ffe5c010499eb9fc20e6702bbd3a (127.11.102.3:42949), b4bfdf3553314427a463f1a1d17da0d3 (127.11.102.2:45705)
I20260620 06:07:06.543895 16165 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dee368d1441e4c64903bbb2bc3590fc7" candidate_uuid: "51a86ef5aaa04847aafbda5c22a02594" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5d07ffe5c010499eb9fc20e6702bbd3a"
I20260620 06:07:06.543962 16032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dee368d1441e4c64903bbb2bc3590fc7" candidate_uuid: "51a86ef5aaa04847aafbda5c22a02594" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b4bfdf3553314427a463f1a1d17da0d3"
I20260620 06:07:06.544018 16165 raft_consensus.cc:3037] T dee368d1441e4c64903bbb2bc3590fc7 P 5d07ffe5c010499eb9fc20e6702bbd3a [term 0 FOLLOWER]: Advancing to term 1
I20260620 06:07:06.544061 16032 raft_consensus.cc:3037] T dee368d1441e4c64903bbb2bc3590fc7 P b4bfdf3553314427a463f1a1d17da0d3 [term 0 FOLLOWER]: Advancing to term 1
I20260620 06:07:06.545295 16032 raft_consensus.cc:2454] T dee368d1441e4c64903bbb2bc3590fc7 P b4bfdf3553314427a463f1a1d17da0d3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 51a86ef5aaa04847aafbda5c22a02594 in term 1.
I20260620 06:07:06.545305 16165 raft_consensus.cc:2454] T dee368d1441e4c64903bbb2bc3590fc7 P 5d07ffe5c010499eb9fc20e6702bbd3a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 51a86ef5aaa04847aafbda5c22a02594 in term 1.
I20260620 06:07:06.545588 15832 leader_election.cc:302] T dee368d1441e4c64903bbb2bc3590fc7 P 51a86ef5aaa04847aafbda5c22a02594 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 51a86ef5aaa04847aafbda5c22a02594, 5d07ffe5c010499eb9fc20e6702bbd3a; no voters:
I20260620 06:07:06.545763 16234 raft_consensus.cc:2781] T dee368d1441e4c64903bbb2bc3590fc7 P 51a86ef5aaa04847aafbda5c22a02594 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 06:07:06.545954 16234 raft_consensus.cc:686] T dee368d1441e4c64903bbb2bc3590fc7 P 51a86ef5aaa04847aafbda5c22a02594 [term 1 LEADER]: Becoming Leader. State: Replica: 51a86ef5aaa04847aafbda5c22a02594, State: Running, Role: LEADER
I20260620 06:07:06.546255 16234 consensus_queue.cc:237] T dee368d1441e4c64903bbb2bc3590fc7 P 51a86ef5aaa04847aafbda5c22a02594 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51a86ef5aaa04847aafbda5c22a02594" member_type: VOTER last_known_addr { host: "127.11.102.1" port: 46757 } } peers { permanent_uuid: "5d07ffe5c010499eb9fc20e6702bbd3a" member_type: VOTER last_known_addr { host: "127.11.102.3" port: 42949 } } peers { permanent_uuid: "b4bfdf3553314427a463f1a1d17da0d3" member_type: VOTER last_known_addr { host: "127.11.102.2" port: 45705 } }
I20260620 06:07:06.549268 15754 catalog_manager.cc:5495] T dee368d1441e4c64903bbb2bc3590fc7 P 51a86ef5aaa04847aafbda5c22a02594 reported cstate change: term changed from 0 to 1, leader changed from <none> to 51a86ef5aaa04847aafbda5c22a02594 (127.11.102.1). New cstate: current_term: 1 leader_uuid: "51a86ef5aaa04847aafbda5c22a02594" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51a86ef5aaa04847aafbda5c22a02594" member_type: VOTER last_known_addr { host: "127.11.102.1" port: 46757 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5d07ffe5c010499eb9fc20e6702bbd3a" member_type: VOTER last_known_addr { host: "127.11.102.3" port: 42949 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b4bfdf3553314427a463f1a1d17da0d3" member_type: VOTER last_known_addr { host: "127.11.102.2" port: 45705 } health_report { overall_health: UNKNOWN } } }
I20260620 06:07:06.597982 16223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:07:06.594779 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49086 (local address 127.11.102.1:46757)
0620 06:07:06.594911 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0620 06:07:06.594915 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 06:07:06.594929 (+ 14us) server_negotiation.cc:407] Connection header received
0620 06:07:06.594997 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:07:06.595000 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:07:06.595049 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:07:06.595135 (+ 86us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 06:07:06.595351 (+ 216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:07:06.595906 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:07:06.596697 (+ 791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:07:06.596890 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:07:06.596964 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:07:06.597201 (+ 237us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 06:07:06.597596 (+ 395us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 06:07:06.597687 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0620 06:07:06.597837 (+ 150us) server_negotiation.cc:299] Negotiation successful
0620 06:07:06.597882 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260620 06:07:06.603411 16032 raft_consensus.cc:1261] T dee368d1441e4c64903bbb2bc3590fc7 P b4bfdf3553314427a463f1a1d17da0d3 [term 1 FOLLOWER]: Refusing update from remote peer 51a86ef5aaa04847aafbda5c22a02594: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 06:07:06.603412 16165 raft_consensus.cc:1261] T dee368d1441e4c64903bbb2bc3590fc7 P 5d07ffe5c010499eb9fc20e6702bbd3a [term 1 FOLLOWER]: Refusing update from remote peer 51a86ef5aaa04847aafbda5c22a02594: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 06:07:06.604090 16239 consensus_queue.cc:1035] T dee368d1441e4c64903bbb2bc3590fc7 P 51a86ef5aaa04847aafbda5c22a02594 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b4bfdf3553314427a463f1a1d17da0d3" member_type: VOTER last_known_addr { host: "127.11.102.2" port: 45705 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 06:07:06.604267 16234 consensus_queue.cc:1035] T dee368d1441e4c64903bbb2bc3590fc7 P 51a86ef5aaa04847aafbda5c22a02594 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5d07ffe5c010499eb9fc20e6702bbd3a" member_type: VOTER last_known_addr { host: "127.11.102.3" port: 42949 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 06:07:06.612059 16244 mvcc.cc:205] Tried to move back new op lower bound from 7298808326561439744 to 7298808326336241664. Current Snapshot: MvccSnapshot[applied={T|T < 7298808326561439744}]
I20260620 06:07:06.614123 16242 mvcc.cc:205] Tried to move back new op lower bound from 7298808326561439744 to 7298808326336241664. Current Snapshot: MvccSnapshot[applied={T|T < 7298808326561439744}]
I20260620 06:07:06.622272 15753 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39994:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 06:07:06.622442 15753 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39994:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 06:07:06.624948 15753 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1afe83e2a69e40d59ff7d0497b27392e: Sending DeleteTablet for 3 replicas of tablet dee368d1441e4c64903bbb2bc3590fc7
I20260620 06:07:06.625662 15879 tablet_service.cc:1514] Processing DeleteTablet for tablet dee368d1441e4c64903bbb2bc3590fc7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 06:07:06 UTC) from {username='kudu', principal='kudu/127.11.102.62@KRBTEST.COM'} at 127.0.0.1:49082
I20260620 06:07:06.625705 16145 tablet_service.cc:1514] Processing DeleteTablet for tablet dee368d1441e4c64903bbb2bc3590fc7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 06:07:06 UTC) from {username='kudu', principal='kudu/127.11.102.62@KRBTEST.COM'} at 127.0.0.1:36494
I20260620 06:07:06.625819 16012 tablet_service.cc:1514] Processing DeleteTablet for tablet dee368d1441e4c64903bbb2bc3590fc7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 06:07:06 UTC) from {username='kudu', principal='kudu/127.11.102.62@KRBTEST.COM'} at 127.0.0.1:43940
I20260620 06:07:06.626092 16252 tablet_replica.cc:330] T dee368d1441e4c64903bbb2bc3590fc7 P 51a86ef5aaa04847aafbda5c22a02594: stopping tablet replica
I20260620 06:07:06.626391 16252 raft_consensus.cc:2229] T dee368d1441e4c64903bbb2bc3590fc7 P 51a86ef5aaa04847aafbda5c22a02594 [term 1 LEADER]: Raft consensus shutting down.
I20260620 06:07:06.626456 16253 tablet_replica.cc:330] T dee368d1441e4c64903bbb2bc3590fc7 P 5d07ffe5c010499eb9fc20e6702bbd3a: stopping tablet replica
I20260620 06:07:06.626753 16252 raft_consensus.cc:2258] T dee368d1441e4c64903bbb2bc3590fc7 P 51a86ef5aaa04847aafbda5c22a02594 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 06:07:06.626965 11672 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9HMp43/build/debug/bin/kudu with pid 15817
I20260620 06:07:06.628605 16254 tablet_replica.cc:330] T dee368d1441e4c64903bbb2bc3590fc7 P b4bfdf3553314427a463f1a1d17da0d3: stopping tablet replica
I20260620 06:07:06.628829 16254 raft_consensus.cc:2229] T dee368d1441e4c64903bbb2bc3590fc7 P b4bfdf3553314427a463f1a1d17da0d3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 06:07:06.629050 16254 raft_consensus.cc:2258] T dee368d1441e4c64903bbb2bc3590fc7 P b4bfdf3553314427a463f1a1d17da0d3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 06:07:06.626719 16253 raft_consensus.cc:2229] T dee368d1441e4c64903bbb2bc3590fc7 P 5d07ffe5c010499eb9fc20e6702bbd3a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 06:07:06.629521 16253 raft_consensus.cc:2258] T dee368d1441e4c64903bbb2bc3590fc7 P 5d07ffe5c010499eb9fc20e6702bbd3a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 06:07:06.630525 16254 ts_tablet_manager.cc:1905] T dee368d1441e4c64903bbb2bc3590fc7 P b4bfdf3553314427a463f1a1d17da0d3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 06:07:06.630654 16253 ts_tablet_manager.cc:1905] T dee368d1441e4c64903bbb2bc3590fc7 P 5d07ffe5c010499eb9fc20e6702bbd3a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 06:07:06.632758 16253 ts_tablet_manager.cc:1918] T dee368d1441e4c64903bbb2bc3590fc7 P 5d07ffe5c010499eb9fc20e6702bbd3a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 06:07:06.632867 16253 log.cc:1196] T dee368d1441e4c64903bbb2bc3590fc7 P 5d07ffe5c010499eb9fc20e6702bbd3a: Deleting WAL directory at /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781935603120653-11672-0/minicluster-data/ts-2/wal/wals/dee368d1441e4c64903bbb2bc3590fc7
I20260620 06:07:06.633148 16253 ts_tablet_manager.cc:1939] T dee368d1441e4c64903bbb2bc3590fc7 P 5d07ffe5c010499eb9fc20e6702bbd3a: Deleting consensus metadata
I20260620 06:07:06.634001 16254 ts_tablet_manager.cc:1918] T dee368d1441e4c64903bbb2bc3590fc7 P b4bfdf3553314427a463f1a1d17da0d3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 06:07:06.634099 16254 log.cc:1196] T dee368d1441e4c64903bbb2bc3590fc7 P b4bfdf3553314427a463f1a1d17da0d3: Deleting WAL directory at /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781935603120653-11672-0/minicluster-data/ts-1/wal/wals/dee368d1441e4c64903bbb2bc3590fc7
I20260620 06:07:06.634267 15739 catalog_manager.cc:4841] TS 5d07ffe5c010499eb9fc20e6702bbd3a (127.11.102.3:42949): tablet dee368d1441e4c64903bbb2bc3590fc7 (table test-table [id=701212e42e554a22bf3c557a9f847696]) successfully deleted
I20260620 06:07:06.634416 16254 ts_tablet_manager.cc:1939] T dee368d1441e4c64903bbb2bc3590fc7 P b4bfdf3553314427a463f1a1d17da0d3: Deleting consensus metadata
W20260620 06:07:06.634712 15742 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.11.102.1:46757 (error 108)
I20260620 06:07:06.635552 15741 catalog_manager.cc:4841] TS b4bfdf3553314427a463f1a1d17da0d3 (127.11.102.2:45705): tablet dee368d1441e4c64903bbb2bc3590fc7 (table test-table [id=701212e42e554a22bf3c557a9f847696]) successfully deleted
I20260620 06:07:06.635675 11672 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9HMp43/build/debug/bin/kudu with pid 15950
I20260620 06:07:06.635713 16225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:07:06.635370 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.102.1:46757 (local address 127.0.0.1:49092)
0620 06:07:06.635492 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0620 06:07:06.635556 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.102.1:46757: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260620 06:07:06.635852 15742 catalog_manager.cc:4584] TS 51a86ef5aaa04847aafbda5c22a02594 (127.11.102.1:46757): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet dee368d1441e4c64903bbb2bc3590fc7: Network error: Client connection negotiation failed: client connection to 127.11.102.1:46757: connect: Connection refused (error 111)
I20260620 06:07:06.642011 11672 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9HMp43/build/debug/bin/kudu with pid 16083
I20260620 06:07:06.647519 11672 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9HMp43/build/debug/bin/kudu with pid 15724
2026-06-20T06:07:06Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (2421 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-06-20T06:07:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T06:07:06Z Disabled control of system clock
I20260620 06:07:06.677304 11672 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9HMp43/build/debug/bin/kudu
/tmp/dist-test-task9HMp43/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781935603120653-11672-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781935603120653-11672-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781935603120653-11672-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781935603120653-11672-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.102.62:43739
--webserver_interface=127.11.102.62
--webserver_port=0
--builtin_ntp_servers=127.11.102.20:43853
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.102.62:43739
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260620 06:07:06.781679 16259 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:07:06.781935 16259 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:07:06.781971 16259 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:07:06.785904 16259 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 06:07:06.785957 16259 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:07:06.785974 16259 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:07:06.785989 16259 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 06:07:06.786005 16259 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 06:07:06.790580 16259 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.102.20:43853
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781935603120653-11672-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781935603120653-11672-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.102.62:43739
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.102.62:43739
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781935603120653-11672-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.102.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16259
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781935603120653-11672-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 05:28:48 UTC on ccycloud.quasar-fixhms.root.comops.site
build id 4457020
I20260620 06:07:06.791596 16259 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:07:06.792940 16259 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 06:07:06.801206 16267 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:07:06.801267 16265 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:07:06.801262 16264 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:07:06.801438 16259 server_base.cc:1034] running on GCE node
I20260620 06:07:06.802079 16259 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:07:06.803184 16259 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:07:06.804368 16259 hybrid_clock.cc:648] HybridClock initialized: now 1781935626804354 us; error 36 us; skew 500 ppm
I20260620 06:07:06.810935 16259 webserver.cc:460] Webserver started at http://127.11.102.62:41625/ using document root <none> and password file <none>
I20260620 06:07:06.811558 16259 fs_manager.cc:359] Metadata directory not provided
I20260620 06:07:06.811681 16259 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:07:06.811923 16259 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 06:07:06.813983 16259 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781935603120653-11672-0/minicluster-data/master-0/data/instance:
uuid: "2a7b97dea0a64ea1b04418276432191a"
format_stamp: "Formatted at 2026-06-20 06:07:06 on dist-test-slave-9hm5"
server_key: "7cd5d0e557411a54f5a3c68fb2bf4b7e"
server_key_iv: "a8af51e1c04b284ae2d4052d5003001b"
server_key_version: "encryptionkey@0"
I20260620 06:07:06.814551 16259 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781935603120653-11672-0/minicluster-data/master-0/wal/instance:
uuid: "2a7b97dea0a64ea1b04418276432191a"
format_stamp: "Formatted at 2026-06-20 06:07:06 on dist-test-slave-9hm5"
server_key: "7cd5d0e557411a54f5a3c68fb2bf4b7e"
server_key_iv: "a8af51e1c04b284ae2d4052d5003001b"
server_key_version: "encryptionkey@0"
I20260620 06:07:06.818189 16259 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.002s
I20260620 06:07:06.820616 16273 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:07:06.821753 16259 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 06:07:06.821890 16259 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781935603120653-11672-0/minicluster-data/master-0/data,/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781935603120653-11672-0/minicluster-data/master-0/wal
uuid: "2a7b97dea0a64ea1b04418276432191a"
format_stamp: "Formatted at 2026-06-20 06:07:06 on dist-test-slave-9hm5"
server_key: "7cd5d0e557411a54f5a3c68fb2bf4b7e"
server_key_iv: "a8af51e1c04b284ae2d4052d5003001b"
server_key_version: "encryptionkey@0"
I20260620 06:07:06.822000 16259 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781935603120653-11672-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781935603120653-11672-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781935603120653-11672-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:07:06.826557 16259 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:07:06.827191 16259 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:07:06.827373 16259 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:07:06.834069 16259 rpc_server.cc:307] RPC server started. Bound to: 127.11.102.62:43739
I20260620 06:07:06.834079 16325 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.102.62:43739 every 8 connection(s)
I20260620 06:07:06.835012 16259 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781935603120653-11672-0/minicluster-data/master-0/data/info.pb
I20260620 06:07:06.838016 16326 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 06:07:06.842079 11672 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9HMp43/build/debug/bin/kudu as pid 16259
I20260620 06:07:06.842209 11672 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781935603120653-11672-0/minicluster-data/master-0/wal/instance
I20260620 06:07:06.842504 11672 external_mini_cluster.cc:1428] Setting key 56fffacf7d6b307edf89eca598956154
I20260620 06:07:06.843693 16326 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2a7b97dea0a64ea1b04418276432191a: Bootstrap starting.
I20260620 06:07:06.845804 16326 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2a7b97dea0a64ea1b04418276432191a: Neither blocks nor log segments found. Creating new log.
I20260620 06:07:06.846684 16326 log.cc:824] T 00000000000000000000000000000000 P 2a7b97dea0a64ea1b04418276432191a: Log is configured to *not* fsync() on all Append() calls
I20260620 06:07:06.847884 16329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:07:06.843968 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50936 (local address 127.11.102.62:43739)
0620 06:07:06.844546 (+ 578us) server_negotiation.cc:206] Beginning negotiation
0620 06:07:06.844562 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0620 06:07:06.844597 (+ 35us) server_negotiation.cc:407] Connection header received
0620 06:07:06.845200 (+ 603us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:07:06.845216 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:07:06.845540 (+ 324us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:07:06.845803 (+ 263us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:07:06.846611 (+ 808us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:07:06.846615 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:07:06.846625 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 06:07:06.846634 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:07:06.846686 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 06:07:06.846701 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:07:06.846818 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0620 06:07:06.846920 (+ 102us) server_negotiation.cc:299] Negotiation successful
0620 06:07:06.847188 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":241,"thread_start_us":125,"threads_started":1}
I20260620 06:07:06.848587 16326 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2a7b97dea0a64ea1b04418276432191a: No bootstrap required, opened a new log
I20260620 06:07:06.851861 16326 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2a7b97dea0a64ea1b04418276432191a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a7b97dea0a64ea1b04418276432191a" member_type: VOTER last_known_addr { host: "127.11.102.62" port: 43739 } }
I20260620 06:07:06.852082 16326 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2a7b97dea0a64ea1b04418276432191a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 06:07:06.852164 16326 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2a7b97dea0a64ea1b04418276432191a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2a7b97dea0a64ea1b04418276432191a, State: Initialized, Role: FOLLOWER
I20260620 06:07:06.852596 16326 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2a7b97dea0a64ea1b04418276432191a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a7b97dea0a64ea1b04418276432191a" member_type: VOTER last_known_addr { host: "127.11.102.62" port: 43739 } }
I20260620 06:07:06.852741 16326 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2a7b97dea0a64ea1b04418276432191a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 06:07:06.852825 16326 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2a7b97dea0a64ea1b04418276432191a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 06:07:06.852919 16326 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2a7b97dea0a64ea1b04418276432191a [term 0 FOLLOWER]: Advancing to term 1
I20260620 06:07:06.853729 16326 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2a7b97dea0a64ea1b04418276432191a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a7b97dea0a64ea1b04418276432191a" member_type: VOTER last_known_addr { host: "127.11.102.62" port: 43739 } }
I20260620 06:07:06.854048 16326 leader_election.cc:302] T 00000000000000000000000000000000 P 2a7b97dea0a64ea1b04418276432191a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2a7b97dea0a64ea1b04418276432191a; no voters:
I20260620 06:07:06.854329 16326 leader_election.cc:288] T 00000000000000000000000000000000 P 2a7b97dea0a64ea1b04418276432191a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 06:07:06.854473 16331 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2a7b97dea0a64ea1b04418276432191a [term 1 FOLLOWER]: Leader election won for term 1
I20260620 06:07:06.854692 16331 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2a7b97dea0a64ea1b04418276432191a [term 1 LEADER]: Becoming Leader. State: Replica: 2a7b97dea0a64ea1b04418276432191a, State: Running, Role: LEADER
I20260620 06:07:06.854983 16331 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2a7b97dea0a64ea1b04418276432191a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a7b97dea0a64ea1b04418276432191a" member_type: VOTER last_known_addr { host: "127.11.102.62" port: 43739 } }
I20260620 06:07:06.855466 16326 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2a7b97dea0a64ea1b04418276432191a [sys.catalog]: configured and running, proceeding with master startup.
I20260620 06:07:06.856534 16332 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2a7b97dea0a64ea1b04418276432191a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2a7b97dea0a64ea1b04418276432191a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a7b97dea0a64ea1b04418276432191a" member_type: VOTER last_known_addr { host: "127.11.102.62" port: 43739 } } }
I20260620 06:07:06.856662 16332 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2a7b97dea0a64ea1b04418276432191a [sys.catalog]: This master's current role is: LEADER
I20260620 06:07:06.856940 16333 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2a7b97dea0a64ea1b04418276432191a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2a7b97dea0a64ea1b04418276432191a. Latest consensus state: current_term: 1 leader_uuid: "2a7b97dea0a64ea1b04418276432191a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a7b97dea0a64ea1b04418276432191a" member_type: VOTER last_known_addr { host: "127.11.102.62" port: 43739 } } }
I20260620 06:07:06.857033 16333 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2a7b97dea0a64ea1b04418276432191a [sys.catalog]: This master's current role is: LEADER
I20260620 06:07:06.857110 16340 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 06:07:06.860236 16340 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 06:07:06.865828 16340 catalog_manager.cc:1347] Generated new cluster ID: 9c1e011652d24456ae7c55b01a80e02f
I20260620 06:07:06.865926 16340 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 06:07:06.875379 16340 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 06:07:06.875734 16340 catalog_manager.cc:1504] Loading token signing keys...
I20260620 06:07:06.891494 16340 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2a7b97dea0a64ea1b04418276432191a: Generated new TSK 0
I20260620 06:07:06.892239 16340 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 06:07:06.898823 11672 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9HMp43/build/debug/bin/kudu
/tmp/dist-test-task9HMp43/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781935603120653-11672-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781935603120653-11672-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781935603120653-11672-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781935603120653-11672-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.102.1:0
--local_ip_for_outbound_sockets=127.11.102.1
--webserver_interface=127.11.102.1
--webserver_port=0
--tserver_master_addrs=127.11.102.62:43739
--builtin_ntp_servers=127.11.102.20:43853
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260620 06:07:07.013450 16350 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:07:07.013703 16350 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:07:07.013743 16350 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:07:07.017594 16350 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:07:07.017637 16350 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:07:07.017688 16350 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.102.1
I20260620 06:07:07.022203 16350 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.102.20:43853
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781935603120653-11672-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781935603120653-11672-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.102.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781935603120653-11672-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.11.102.1
--webserver_port=0
--tserver_master_addrs=127.11.102.62:43739
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16350
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.102.1
--log_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781935603120653-11672-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 05:28:48 UTC on ccycloud.quasar-fixhms.root.comops.site
build id 4457020
I20260620 06:07:07.023070 16350 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:07:07.024309 16350 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 06:07:07.033589 16356 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:07:07.033601 16358 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:07:07.033601 16355 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:07:07.033821 16350 server_base.cc:1034] running on GCE node
I20260620 06:07:07.034287 16350 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:07:07.034857 16350 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:07:07.036032 16350 hybrid_clock.cc:648] HybridClock initialized: now 1781935627036019 us; error 31 us; skew 500 ppm
I20260620 06:07:07.037627 16350 webserver.cc:460] Webserver started at http://127.11.102.1:41957/ using document root <none> and password file <none>
I20260620 06:07:07.038172 16350 fs_manager.cc:359] Metadata directory not provided
I20260620 06:07:07.038259 16350 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:07:07.038476 16350 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 06:07:07.040413 16350 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781935603120653-11672-0/minicluster-data/ts-0/data/instance:
uuid: "105794c45c6b461f826ca2ff3ee487a3"
format_stamp: "Formatted at 2026-06-20 06:07:07 on dist-test-slave-9hm5"
server_key: "1ed3460a62f05bf8dbfdd6eabb8b9eb1"
server_key_iv: "52004ad10da0d4e713b0a2a6ed035513"
server_key_version: "encryptionkey@0"
I20260620 06:07:07.040978 16350 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781935603120653-11672-0/minicluster-data/ts-0/wal/instance:
uuid: "105794c45c6b461f826ca2ff3ee487a3"
format_stamp: "Formatted at 2026-06-20 06:07:07 on dist-test-slave-9hm5"
server_key: "1ed3460a62f05bf8dbfdd6eabb8b9eb1"
server_key_iv: "52004ad10da0d4e713b0a2a6ed035513"
server_key_version: "encryptionkey@0"
I20260620 06:07:07.044368 16350 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 06:07:07.046662 16364 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:07:07.047875 16350 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 06:07:07.048012 16350 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781935603120653-11672-0/minicluster-data/ts-0/data,/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781935603120653-11672-0/minicluster-data/ts-0/wal
uuid: "105794c45c6b461f826ca2ff3ee487a3"
format_stamp: "Formatted at 2026-06-20 06:07:07 on dist-test-slave-9hm5"
server_key: "1ed3460a62f05bf8dbfdd6eabb8b9eb1"
server_key_iv: "52004ad10da0d4e713b0a2a6ed035513"
server_key_version: "encryptionkey@0"
I20260620 06:07:07.048116 16350 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781935603120653-11672-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781935603120653-11672-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781935603120653-11672-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:07:07.052425 16350 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:07:07.052977 16350 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:07:07.053161 16350 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:07:07.053670 16350 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 06:07:07.054510 16350 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 06:07:07.054579 16350 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 06:07:07.054638 16350 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 06:07:07.054689 16350 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 06:07:07.063571 16350 rpc_server.cc:307] RPC server started. Bound to: 127.11.102.1:44995
I20260620 06:07:07.063621 16477 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.102.1:44995 every 8 connection(s)
I20260620 06:07:07.064409 16350 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781935603120653-11672-0/minicluster-data/ts-0/data/info.pb
I20260620 06:07:07.065191 11672 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9HMp43/build/debug/bin/kudu as pid 16350
I20260620 06:07:07.065308 11672 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781935603120653-11672-0/minicluster-data/ts-0/wal/instance
I20260620 06:07:07.065603 11672 external_mini_cluster.cc:1428] Setting key 34f96c2048da71d2f1d7fcc091a1b49b
I20260620 06:07:07.067874 11672 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9HMp43/build/debug/bin/kudu
/tmp/dist-test-task9HMp43/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781935603120653-11672-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781935603120653-11672-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781935603120653-11672-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781935603120653-11672-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.102.2:0
--local_ip_for_outbound_sockets=127.11.102.2
--webserver_interface=127.11.102.2
--webserver_port=0
--tserver_master_addrs=127.11.102.62:43739
--builtin_ntp_servers=127.11.102.20:43853
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260620 06:07:07.071002 16329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:07:07.066255 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.102.1:44997 (local address 127.11.102.62:43739)
0620 06:07:07.066411 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0620 06:07:07.066414 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 06:07:07.067745 (+ 1331us) server_negotiation.cc:407] Connection header received
0620 06:07:07.069315 (+ 1570us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:07:07.069319 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:07:07.069356 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:07:07.069431 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:07:07.070565 (+ 1134us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:07:07.070568 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:07:07.070570 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 06:07:07.070581 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:07:07.070600 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 06:07:07.070609 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:07:07.070686 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0620 06:07:07.070851 (+ 165us) server_negotiation.cc:299] Negotiation successful
0620 06:07:07.070884 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260620 06:07:07.072124 16480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:07:07.066772 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.102.62:43739 (local address 127.11.102.1:44997)
0620 06:07:07.067563 (+ 791us) negotiation.cc:107] Waiting for socket to connect
0620 06:07:07.067600 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0620 06:07:07.069124 (+ 1524us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:07:07.069583 (+ 459us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:07:07.069589 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:07:07.069671 (+ 82us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:07:07.070269 (+ 598us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 06:07:07.070273 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:07:07.070326 (+ 53us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 06:07:07.070414 (+ 88us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:07:07.070700 (+ 286us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:07:07.070711 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:07:07.070719 (+ 8us) client_negotiation.cc:769] Sending connection context
0620 06:07:07.070883 (+ 164us) client_negotiation.cc:240] Negotiation successful
0620 06:07:07.071114 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":378,"thread_start_us":144,"threads_started":1}
I20260620 06:07:07.073341 16478 heartbeater.cc:347] Connected to a master server at 127.11.102.62:43739
I20260620 06:07:07.073553 16478 heartbeater.cc:464] Registering TS with master...
I20260620 06:07:07.073945 16478 heartbeater.cc:511] Master 127.11.102.62:43739 requested a full tablet report, sending...
I20260620 06:07:07.075690 16290 ts_manager.cc:194] Registered new tserver with Master: 105794c45c6b461f826ca2ff3ee487a3 (127.11.102.1:44995)
W20260620 06:07:07.174772 16481 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:07:07.175007 16481 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:07:07.175043 16481 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:07:07.178905 16481 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:07:07.178944 16481 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:07:07.179006 16481 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.102.2
I20260620 06:07:07.183357 16481 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.102.20:43853
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781935603120653-11672-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781935603120653-11672-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.102.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781935603120653-11672-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.11.102.2
--webserver_port=0
--tserver_master_addrs=127.11.102.62:43739
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16481
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.102.2
--log_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781935603120653-11672-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 05:28:48 UTC on ccycloud.quasar-fixhms.root.comops.site
build id 4457020
I20260620 06:07:07.184227 16481 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:07:07.185388 16481 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 06:07:07.194764 16489 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:07:07.194725 16487 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:07:07.194867 16481 server_base.cc:1034] running on GCE node
W20260620 06:07:07.194717 16486 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:07:07.195364 16481 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:07:07.196025 16481 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:07:07.197181 16481 hybrid_clock.cc:648] HybridClock initialized: now 1781935627197165 us; error 28 us; skew 500 ppm
I20260620 06:07:07.198936 16481 webserver.cc:460] Webserver started at http://127.11.102.2:39839/ using document root <none> and password file <none>
I20260620 06:07:07.199474 16481 fs_manager.cc:359] Metadata directory not provided
I20260620 06:07:07.199533 16481 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:07:07.199787 16481 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 06:07:07.201684 16481 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781935603120653-11672-0/minicluster-data/ts-1/data/instance:
uuid: "ecaa1724bb774d869b5edc0f9cfa49c3"
format_stamp: "Formatted at 2026-06-20 06:07:07 on dist-test-slave-9hm5"
server_key: "16a11a5e99229c31448e6510883d4d8b"
server_key_iv: "c3b8051de1761fee75685ac348822f40"
server_key_version: "encryptionkey@0"
I20260620 06:07:07.202229 16481 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781935603120653-11672-0/minicluster-data/ts-1/wal/instance:
uuid: "ecaa1724bb774d869b5edc0f9cfa49c3"
format_stamp: "Formatted at 2026-06-20 06:07:07 on dist-test-slave-9hm5"
server_key: "16a11a5e99229c31448e6510883d4d8b"
server_key_iv: "c3b8051de1761fee75685ac348822f40"
server_key_version: "encryptionkey@0"
I20260620 06:07:07.205706 16481 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.004s
I20260620 06:07:07.208057 16495 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:07:07.209208 16481 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 06:07:07.209297 16481 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781935603120653-11672-0/minicluster-data/ts-1/data,/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781935603120653-11672-0/minicluster-data/ts-1/wal
uuid: "ecaa1724bb774d869b5edc0f9cfa49c3"
format_stamp: "Formatted at 2026-06-20 06:07:07 on dist-test-slave-9hm5"
server_key: "16a11a5e99229c31448e6510883d4d8b"
server_key_iv: "c3b8051de1761fee75685ac348822f40"
server_key_version: "encryptionkey@0"
I20260620 06:07:07.209429 16481 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781935603120653-11672-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781935603120653-11672-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781935603120653-11672-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:07:07.213965 16481 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:07:07.214509 16481 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:07:07.214679 16481 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:07:07.215176 16481 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 06:07:07.216075 16481 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 06:07:07.216120 16481 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 06:07:07.216183 16481 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 06:07:07.216223 16481 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 06:07:07.224411 16481 rpc_server.cc:307] RPC server started. Bound to: 127.11.102.2:42361
I20260620 06:07:07.224436 16608 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.102.2:42361 every 8 connection(s)
I20260620 06:07:07.225229 16481 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781935603120653-11672-0/minicluster-data/ts-1/data/info.pb
I20260620 06:07:07.230950 16329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:07:07.227000 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.102.2:52647 (local address 127.11.102.62:43739)
0620 06:07:07.227126 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0620 06:07:07.227130 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 06:07:07.228171 (+ 1041us) server_negotiation.cc:407] Connection header received
0620 06:07:07.229364 (+ 1193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:07:07.229367 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:07:07.229411 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:07:07.229485 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:07:07.230578 (+ 1093us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:07:07.230581 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:07:07.230583 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 06:07:07.230595 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:07:07.230613 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 06:07:07.230620 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:07:07.230690 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0620 06:07:07.230810 (+ 120us) server_negotiation.cc:299] Negotiation successful
0620 06:07:07.230845 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260620 06:07:07.231851 16611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:07:07.227346 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.102.62:43739 (local address 127.11.102.2:52647)
0620 06:07:07.228023 (+ 677us) negotiation.cc:107] Waiting for socket to connect
0620 06:07:07.228061 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0620 06:07:07.229181 (+ 1120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:07:07.229660 (+ 479us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:07:07.229668 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:07:07.229777 (+ 109us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:07:07.230344 (+ 567us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 06:07:07.230346 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:07:07.230381 (+ 35us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 06:07:07.230444 (+ 63us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:07:07.230692 (+ 248us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:07:07.230698 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:07:07.230702 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 06:07:07.230822 (+ 120us) client_negotiation.cc:240] Negotiation successful
0620 06:07:07.231053 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":280,"thread_start_us":103,"threads_started":1}
I20260620 06:07:07.232864 16609 heartbeater.cc:347] Connected to a master server at 127.11.102.62:43739
I20260620 06:07:07.233103 16609 heartbeater.cc:464] Registering TS with master...
I20260620 06:07:07.233476 16609 heartbeater.cc:511] Master 127.11.102.62:43739 requested a full tablet report, sending...
I20260620 06:07:07.234071 11672 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9HMp43/build/debug/bin/kudu as pid 16481
I20260620 06:07:07.234193 11672 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781935603120653-11672-0/minicluster-data/ts-1/wal/instance
I20260620 06:07:07.234483 11672 external_mini_cluster.cc:1428] Setting key 3c8b3074b308b61b6ea44f3aa21767a1
I20260620 06:07:07.234534 16290 ts_manager.cc:194] Registered new tserver with Master: ecaa1724bb774d869b5edc0f9cfa49c3 (127.11.102.2:42361)
I20260620 06:07:07.236404 11672 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9HMp43/build/debug/bin/kudu
/tmp/dist-test-task9HMp43/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781935603120653-11672-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781935603120653-11672-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781935603120653-11672-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781935603120653-11672-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.102.3:0
--local_ip_for_outbound_sockets=127.11.102.3
--webserver_interface=127.11.102.3
--webserver_port=0
--tserver_master_addrs=127.11.102.62:43739
--builtin_ntp_servers=127.11.102.20:43853
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260620 06:07:07.341905 16612 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:07:07.342142 16612 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:07:07.342180 16612 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:07:07.345978 16612 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:07:07.346026 16612 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:07:07.346079 16612 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.102.3
I20260620 06:07:07.350490 16612 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.102.20:43853
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781935603120653-11672-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781935603120653-11672-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.102.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781935603120653-11672-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.11.102.3
--webserver_port=0
--tserver_master_addrs=127.11.102.62:43739
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16612
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.102.3
--log_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781935603120653-11672-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 05:28:48 UTC on ccycloud.quasar-fixhms.root.comops.site
build id 4457020
I20260620 06:07:07.351472 16612 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:07:07.352844 16612 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 06:07:07.362156 16620 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:07:07.362200 16617 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:07:07.362239 16618 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:07:07.362217 16612 server_base.cc:1034] running on GCE node
I20260620 06:07:07.362717 16612 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:07:07.363303 16612 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:07:07.364480 16612 hybrid_clock.cc:648] HybridClock initialized: now 1781935627364466 us; error 29 us; skew 500 ppm
I20260620 06:07:07.366227 16612 webserver.cc:460] Webserver started at http://127.11.102.3:34457/ using document root <none> and password file <none>
I20260620 06:07:07.366772 16612 fs_manager.cc:359] Metadata directory not provided
I20260620 06:07:07.366865 16612 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:07:07.367090 16612 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 06:07:07.369052 16612 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781935603120653-11672-0/minicluster-data/ts-2/data/instance:
uuid: "08dc04cdd71a4f44863ac7eee140a55e"
format_stamp: "Formatted at 2026-06-20 06:07:07 on dist-test-slave-9hm5"
server_key: "d9c4534cfbaa7715a6833b456e9294af"
server_key_iv: "603d76a42351d2f5991d884ce953bbb3"
server_key_version: "encryptionkey@0"
I20260620 06:07:07.369588 16612 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781935603120653-11672-0/minicluster-data/ts-2/wal/instance:
uuid: "08dc04cdd71a4f44863ac7eee140a55e"
format_stamp: "Formatted at 2026-06-20 06:07:07 on dist-test-slave-9hm5"
server_key: "d9c4534cfbaa7715a6833b456e9294af"
server_key_iv: "603d76a42351d2f5991d884ce953bbb3"
server_key_version: "encryptionkey@0"
I20260620 06:07:07.373011 16612 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 06:07:07.375375 16626 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:07:07.376592 16612 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 06:07:07.376722 16612 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781935603120653-11672-0/minicluster-data/ts-2/data,/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781935603120653-11672-0/minicluster-data/ts-2/wal
uuid: "08dc04cdd71a4f44863ac7eee140a55e"
format_stamp: "Formatted at 2026-06-20 06:07:07 on dist-test-slave-9hm5"
server_key: "d9c4534cfbaa7715a6833b456e9294af"
server_key_iv: "603d76a42351d2f5991d884ce953bbb3"
server_key_version: "encryptionkey@0"
I20260620 06:07:07.376847 16612 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781935603120653-11672-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781935603120653-11672-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781935603120653-11672-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:07:07.381239 16612 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:07:07.381819 16612 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:07:07.381999 16612 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:07:07.382509 16612 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 06:07:07.383361 16612 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 06:07:07.383430 16612 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 06:07:07.383492 16612 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 06:07:07.383539 16612 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 06:07:07.392014 16612 rpc_server.cc:307] RPC server started. Bound to: 127.11.102.3:43869
I20260620 06:07:07.392046 16739 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.102.3:43869 every 8 connection(s)
I20260620 06:07:07.392830 16612 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781935603120653-11672-0/minicluster-data/ts-2/data/info.pb
I20260620 06:07:07.398460 16329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:07:07.394504 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.102.3:46773 (local address 127.11.102.62:43739)
0620 06:07:07.394661 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0620 06:07:07.394666 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 06:07:07.395664 (+ 998us) server_negotiation.cc:407] Connection header received
0620 06:07:07.396908 (+ 1244us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:07:07.396911 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:07:07.396946 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:07:07.397022 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:07:07.398002 (+ 980us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:07:07.398005 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:07:07.398007 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 06:07:07.398018 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:07:07.398036 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 06:07:07.398044 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:07:07.398119 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0620 06:07:07.398329 (+ 210us) server_negotiation.cc:299] Negotiation successful
0620 06:07:07.398363 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260620 06:07:07.399328 16742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:07:07.394903 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.102.62:43739 (local address 127.11.102.3:46773)
0620 06:07:07.395489 (+ 586us) negotiation.cc:107] Waiting for socket to connect
0620 06:07:07.395526 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0620 06:07:07.396696 (+ 1170us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:07:07.397170 (+ 474us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:07:07.397177 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:07:07.397258 (+ 81us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:07:07.397765 (+ 507us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 06:07:07.397767 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:07:07.397804 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 06:07:07.397867 (+ 63us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:07:07.398135 (+ 268us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:07:07.398143 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:07:07.398148 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 06:07:07.398359 (+ 211us) client_negotiation.cc:240] Negotiation successful
0620 06:07:07.398588 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":221,"thread_start_us":93,"threads_started":1}
I20260620 06:07:07.400483 16740 heartbeater.cc:347] Connected to a master server at 127.11.102.62:43739
I20260620 06:07:07.400676 16740 heartbeater.cc:464] Registering TS with master...
I20260620 06:07:07.401010 16740 heartbeater.cc:511] Master 127.11.102.62:43739 requested a full tablet report, sending...
I20260620 06:07:07.401822 11672 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9HMp43/build/debug/bin/kudu as pid 16612
I20260620 06:07:07.401950 11672 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781935603120653-11672-0/minicluster-data/ts-2/wal/instance
I20260620 06:07:07.402022 16290 ts_manager.cc:194] Registered new tserver with Master: 08dc04cdd71a4f44863ac7eee140a55e (127.11.102.3:43869)
I20260620 06:07:07.402256 11672 external_mini_cluster.cc:1428] Setting key f3ee7966d1805d3f8ca9116f44b8be85
I20260620 06:07:07.404162 11672 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 06:07:07.407371 16329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:07:07.406016 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50938 (local address 127.11.102.62:43739)
0620 06:07:07.406135 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0620 06:07:07.406139 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 06:07:07.406302 (+ 163us) server_negotiation.cc:407] Connection header received
0620 06:07:07.406388 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:07:07.406390 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:07:07.406416 (+ 26us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:07:07.406477 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:07:07.407121 (+ 644us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:07:07.407123 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:07:07.407125 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 06:07:07.407131 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:07:07.407143 (+ 12us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 06:07:07.407149 (+ 6us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:07:07.407209 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0620 06:07:07.407250 (+ 41us) server_negotiation.cc:299] Negotiation successful
0620 06:07:07.407277 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260620 06:07:07.410085 16290 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50938:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 06:07:07.413087 16290 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 06:07:07.424755 16752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:07:07.421569 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34270 (local address 127.11.102.3:43869)
0620 06:07:07.421937 (+ 368us) server_negotiation.cc:206] Beginning negotiation
0620 06:07:07.421949 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 06:07:07.421975 (+ 26us) server_negotiation.cc:407] Connection header received
0620 06:07:07.422032 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:07:07.422040 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:07:07.422240 (+ 200us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:07:07.422400 (+ 160us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:07:07.424159 (+ 1759us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:07:07.424166 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:07:07.424178 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 06:07:07.424190 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:07:07.424231 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 06:07:07.424264 (+ 33us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:07:07.424371 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0620 06:07:07.424553 (+ 182us) server_negotiation.cc:299] Negotiation successful
0620 06:07:07.424621 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":271,"thread_start_us":154,"threads_started":1}
I20260620 06:07:07.424728 16753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:07:07.422430 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.102.1:44995 (local address 127.0.0.1:52814)
0620 06:07:07.423141 (+ 711us) negotiation.cc:107] Waiting for socket to connect
0620 06:07:07.423153 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 06:07:07.423227 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:07:07.423699 (+ 472us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:07:07.423702 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:07:07.423716 (+ 14us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:07:07.424108 (+ 392us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 06:07:07.424110 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:07:07.424121 (+ 11us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 06:07:07.424148 (+ 27us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:07:07.424460 (+ 312us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:07:07.424463 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:07:07.424466 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 06:07:07.424550 (+ 84us) client_negotiation.cc:240] Negotiation successful
0620 06:07:07.424586 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":656,"thread_start_us":91,"threads_started":1}
I20260620 06:07:07.424739 16755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:07:07.422114 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.102.2:42361 (local address 127.0.0.1:41146)
0620 06:07:07.422656 (+ 542us) negotiation.cc:107] Waiting for socket to connect
0620 06:07:07.422669 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 06:07:07.422739 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:07:07.423222 (+ 483us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:07:07.423225 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:07:07.423241 (+ 16us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:07:07.424120 (+ 879us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 06:07:07.424121 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:07:07.424128 (+ 7us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 06:07:07.424148 (+ 20us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:07:07.424510 (+ 362us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:07:07.424513 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:07:07.424517 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 06:07:07.424562 (+ 45us) client_negotiation.cc:240] Negotiation successful
0620 06:07:07.424591 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":475,"mutex_wait_us":1,"thread_start_us":85,"threads_started":1}
I20260620 06:07:07.425202 16754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:07:07.422258 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41146 (local address 127.11.102.2:42361)
0620 06:07:07.422757 (+ 499us) server_negotiation.cc:206] Beginning negotiation
0620 06:07:07.422771 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 06:07:07.422806 (+ 35us) server_negotiation.cc:407] Connection header received
0620 06:07:07.422857 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:07:07.422862 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:07:07.423055 (+ 193us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:07:07.423231 (+ 176us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:07:07.424278 (+ 1047us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:07:07.424284 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:07:07.424297 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 06:07:07.424335 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:07:07.424379 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 06:07:07.424410 (+ 31us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:07:07.424932 (+ 522us) server_negotiation.cc:1035] Waiting for connection context
0620 06:07:07.424978 (+ 46us) server_negotiation.cc:299] Negotiation successful
0620 06:07:07.425044 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":401,"thread_start_us":105,"threads_started":1}
I20260620 06:07:07.425351 16756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:07:07.422808 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52814 (local address 127.11.102.1:44995)
0620 06:07:07.423122 (+ 314us) server_negotiation.cc:206] Beginning negotiation
0620 06:07:07.423134 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 06:07:07.423335 (+ 201us) server_negotiation.cc:407] Connection header received
0620 06:07:07.423394 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:07:07.423401 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:07:07.423550 (+ 149us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:07:07.423716 (+ 166us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:07:07.424295 (+ 579us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:07:07.424303 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:07:07.424315 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 06:07:07.424327 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:07:07.424363 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 06:07:07.424390 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:07:07.424479 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0620 06:07:07.425126 (+ 647us) server_negotiation.cc:299] Negotiation successful
0620 06:07:07.425193 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":213,"thread_start_us":95,"threads_started":1}
I20260620 06:07:07.425582 16751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:07:07.421454 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.102.3:43869 (local address 127.0.0.1:34270)
0620 06:07:07.421759 (+ 305us) negotiation.cc:107] Waiting for socket to connect
0620 06:07:07.421810 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0620 06:07:07.421905 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:07:07.423102 (+ 1197us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:07:07.423109 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:07:07.423190 (+ 81us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:07:07.423922 (+ 732us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 06:07:07.423924 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:07:07.423944 (+ 20us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 06:07:07.424016 (+ 72us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:07:07.424358 (+ 342us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:07:07.424369 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:07:07.424375 (+ 6us) client_negotiation.cc:769] Sending connection context
0620 06:07:07.424490 (+ 115us) client_negotiation.cc:240] Negotiation successful
0620 06:07:07.424559 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":69,"threads_started":1}
I20260620 06:07:07.428256 16674 tablet_service.cc:1467] Processing CreateTablet for tablet 9d78abde8d8d43a1846793bc1af40fa5 (DEFAULT_TABLE table=test-table [id=1225e00e238d4c6c979d3f663ddaa298]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 06:07:07.428264 16543 tablet_service.cc:1467] Processing CreateTablet for tablet 9d78abde8d8d43a1846793bc1af40fa5 (DEFAULT_TABLE table=test-table [id=1225e00e238d4c6c979d3f663ddaa298]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 06:07:07.429370 16674 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9d78abde8d8d43a1846793bc1af40fa5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 06:07:07.429370 16543 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9d78abde8d8d43a1846793bc1af40fa5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 06:07:07.429553 16412 tablet_service.cc:1467] Processing CreateTablet for tablet 9d78abde8d8d43a1846793bc1af40fa5 (DEFAULT_TABLE table=test-table [id=1225e00e238d4c6c979d3f663ddaa298]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 06:07:07.430397 16412 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9d78abde8d8d43a1846793bc1af40fa5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 06:07:07.436039 16757 tablet_bootstrap.cc:492] T 9d78abde8d8d43a1846793bc1af40fa5 P 105794c45c6b461f826ca2ff3ee487a3: Bootstrap starting.
I20260620 06:07:07.437016 16758 tablet_bootstrap.cc:492] T 9d78abde8d8d43a1846793bc1af40fa5 P 08dc04cdd71a4f44863ac7eee140a55e: Bootstrap starting.
I20260620 06:07:07.438620 16757 tablet_bootstrap.cc:654] T 9d78abde8d8d43a1846793bc1af40fa5 P 105794c45c6b461f826ca2ff3ee487a3: Neither blocks nor log segments found. Creating new log.
I20260620 06:07:07.438995 16758 tablet_bootstrap.cc:654] T 9d78abde8d8d43a1846793bc1af40fa5 P 08dc04cdd71a4f44863ac7eee140a55e: Neither blocks nor log segments found. Creating new log.
I20260620 06:07:07.439386 16757 log.cc:824] T 9d78abde8d8d43a1846793bc1af40fa5 P 105794c45c6b461f826ca2ff3ee487a3: Log is configured to *not* fsync() on all Append() calls
I20260620 06:07:07.439776 16758 log.cc:824] T 9d78abde8d8d43a1846793bc1af40fa5 P 08dc04cdd71a4f44863ac7eee140a55e: Log is configured to *not* fsync() on all Append() calls
I20260620 06:07:07.440122 16759 tablet_bootstrap.cc:492] T 9d78abde8d8d43a1846793bc1af40fa5 P ecaa1724bb774d869b5edc0f9cfa49c3: Bootstrap starting.
I20260620 06:07:07.441267 16757 tablet_bootstrap.cc:492] T 9d78abde8d8d43a1846793bc1af40fa5 P 105794c45c6b461f826ca2ff3ee487a3: No bootstrap required, opened a new log
I20260620 06:07:07.441488 16757 ts_tablet_manager.cc:1397] T 9d78abde8d8d43a1846793bc1af40fa5 P 105794c45c6b461f826ca2ff3ee487a3: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260620 06:07:07.441675 16758 tablet_bootstrap.cc:492] T 9d78abde8d8d43a1846793bc1af40fa5 P 08dc04cdd71a4f44863ac7eee140a55e: No bootstrap required, opened a new log
I20260620 06:07:07.441864 16758 ts_tablet_manager.cc:1397] T 9d78abde8d8d43a1846793bc1af40fa5 P 08dc04cdd71a4f44863ac7eee140a55e: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.003s
I20260620 06:07:07.442375 16759 tablet_bootstrap.cc:654] T 9d78abde8d8d43a1846793bc1af40fa5 P ecaa1724bb774d869b5edc0f9cfa49c3: Neither blocks nor log segments found. Creating new log.
I20260620 06:07:07.443156 16759 log.cc:824] T 9d78abde8d8d43a1846793bc1af40fa5 P ecaa1724bb774d869b5edc0f9cfa49c3: Log is configured to *not* fsync() on all Append() calls
I20260620 06:07:07.445189 16759 tablet_bootstrap.cc:492] T 9d78abde8d8d43a1846793bc1af40fa5 P ecaa1724bb774d869b5edc0f9cfa49c3: No bootstrap required, opened a new log
I20260620 06:07:07.445379 16759 ts_tablet_manager.cc:1397] T 9d78abde8d8d43a1846793bc1af40fa5 P ecaa1724bb774d869b5edc0f9cfa49c3: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260620 06:07:07.445465 16757 raft_consensus.cc:348] T 9d78abde8d8d43a1846793bc1af40fa5 P 105794c45c6b461f826ca2ff3ee487a3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08dc04cdd71a4f44863ac7eee140a55e" member_type: VOTER last_known_addr { host: "127.11.102.3" port: 43869 } } peers { permanent_uuid: "ecaa1724bb774d869b5edc0f9cfa49c3" member_type: VOTER last_known_addr { host: "127.11.102.2" port: 42361 } } peers { permanent_uuid: "105794c45c6b461f826ca2ff3ee487a3" member_type: VOTER last_known_addr { host: "127.11.102.1" port: 44995 } }
I20260620 06:07:07.445717 16757 raft_consensus.cc:374] T 9d78abde8d8d43a1846793bc1af40fa5 P 105794c45c6b461f826ca2ff3ee487a3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 06:07:07.445780 16757 raft_consensus.cc:729] T 9d78abde8d8d43a1846793bc1af40fa5 P 105794c45c6b461f826ca2ff3ee487a3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 105794c45c6b461f826ca2ff3ee487a3, State: Initialized, Role: FOLLOWER
I20260620 06:07:07.446220 16757 consensus_queue.cc:260] T 9d78abde8d8d43a1846793bc1af40fa5 P 105794c45c6b461f826ca2ff3ee487a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08dc04cdd71a4f44863ac7eee140a55e" member_type: VOTER last_known_addr { host: "127.11.102.3" port: 43869 } } peers { permanent_uuid: "ecaa1724bb774d869b5edc0f9cfa49c3" member_type: VOTER last_known_addr { host: "127.11.102.2" port: 42361 } } peers { permanent_uuid: "105794c45c6b461f826ca2ff3ee487a3" member_type: VOTER last_known_addr { host: "127.11.102.1" port: 44995 } }
I20260620 06:07:07.446149 16758 raft_consensus.cc:348] T 9d78abde8d8d43a1846793bc1af40fa5 P 08dc04cdd71a4f44863ac7eee140a55e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08dc04cdd71a4f44863ac7eee140a55e" member_type: VOTER last_known_addr { host: "127.11.102.3" port: 43869 } } peers { permanent_uuid: "ecaa1724bb774d869b5edc0f9cfa49c3" member_type: VOTER last_known_addr { host: "127.11.102.2" port: 42361 } } peers { permanent_uuid: "105794c45c6b461f826ca2ff3ee487a3" member_type: VOTER last_known_addr { host: "127.11.102.1" port: 44995 } }
I20260620 06:07:07.446380 16758 raft_consensus.cc:374] T 9d78abde8d8d43a1846793bc1af40fa5 P 08dc04cdd71a4f44863ac7eee140a55e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 06:07:07.446434 16758 raft_consensus.cc:729] T 9d78abde8d8d43a1846793bc1af40fa5 P 08dc04cdd71a4f44863ac7eee140a55e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 08dc04cdd71a4f44863ac7eee140a55e, State: Initialized, Role: FOLLOWER
I20260620 06:07:07.446851 16758 consensus_queue.cc:260] T 9d78abde8d8d43a1846793bc1af40fa5 P 08dc04cdd71a4f44863ac7eee140a55e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08dc04cdd71a4f44863ac7eee140a55e" member_type: VOTER last_known_addr { host: "127.11.102.3" port: 43869 } } peers { permanent_uuid: "ecaa1724bb774d869b5edc0f9cfa49c3" member_type: VOTER last_known_addr { host: "127.11.102.2" port: 42361 } } peers { permanent_uuid: "105794c45c6b461f826ca2ff3ee487a3" member_type: VOTER last_known_addr { host: "127.11.102.1" port: 44995 } }
I20260620 06:07:07.447162 16757 ts_tablet_manager.cc:1428] T 9d78abde8d8d43a1846793bc1af40fa5 P 105794c45c6b461f826ca2ff3ee487a3: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260620 06:07:07.447499 16478 heartbeater.cc:503] Master 127.11.102.62:43739 was elected leader, sending a full tablet report...
I20260620 06:07:07.447499 16740 heartbeater.cc:503] Master 127.11.102.62:43739 was elected leader, sending a full tablet report...
I20260620 06:07:07.447791 16758 ts_tablet_manager.cc:1428] T 9d78abde8d8d43a1846793bc1af40fa5 P 08dc04cdd71a4f44863ac7eee140a55e: Time spent starting tablet: real 0.006s user 0.005s sys 0.002s
I20260620 06:07:07.449534 16759 raft_consensus.cc:348] T 9d78abde8d8d43a1846793bc1af40fa5 P ecaa1724bb774d869b5edc0f9cfa49c3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08dc04cdd71a4f44863ac7eee140a55e" member_type: VOTER last_known_addr { host: "127.11.102.3" port: 43869 } } peers { permanent_uuid: "ecaa1724bb774d869b5edc0f9cfa49c3" member_type: VOTER last_known_addr { host: "127.11.102.2" port: 42361 } } peers { permanent_uuid: "105794c45c6b461f826ca2ff3ee487a3" member_type: VOTER last_known_addr { host: "127.11.102.1" port: 44995 } }
I20260620 06:07:07.449776 16759 raft_consensus.cc:374] T 9d78abde8d8d43a1846793bc1af40fa5 P ecaa1724bb774d869b5edc0f9cfa49c3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 06:07:07.449855 16759 raft_consensus.cc:729] T 9d78abde8d8d43a1846793bc1af40fa5 P ecaa1724bb774d869b5edc0f9cfa49c3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ecaa1724bb774d869b5edc0f9cfa49c3, State: Initialized, Role: FOLLOWER
I20260620 06:07:07.450287 16759 consensus_queue.cc:260] T 9d78abde8d8d43a1846793bc1af40fa5 P ecaa1724bb774d869b5edc0f9cfa49c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08dc04cdd71a4f44863ac7eee140a55e" member_type: VOTER last_known_addr { host: "127.11.102.3" port: 43869 } } peers { permanent_uuid: "ecaa1724bb774d869b5edc0f9cfa49c3" member_type: VOTER last_known_addr { host: "127.11.102.2" port: 42361 } } peers { permanent_uuid: "105794c45c6b461f826ca2ff3ee487a3" member_type: VOTER last_known_addr { host: "127.11.102.1" port: 44995 } }
I20260620 06:07:07.450909 16609 heartbeater.cc:503] Master 127.11.102.62:43739 was elected leader, sending a full tablet report...
I20260620 06:07:07.451216 16759 ts_tablet_manager.cc:1428] T 9d78abde8d8d43a1846793bc1af40fa5 P ecaa1724bb774d869b5edc0f9cfa49c3: Time spent starting tablet: real 0.006s user 0.000s sys 0.007s
W20260620 06:07:07.476239 16610 tablet.cc:2369] T 9d78abde8d8d43a1846793bc1af40fa5 P ecaa1724bb774d869b5edc0f9cfa49c3: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 06:07:07.565480 16479 tablet.cc:2369] T 9d78abde8d8d43a1846793bc1af40fa5 P 105794c45c6b461f826ca2ff3ee487a3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 06:07:07.572968 16763 raft_consensus.cc:482] T 9d78abde8d8d43a1846793bc1af40fa5 P 105794c45c6b461f826ca2ff3ee487a3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 06:07:07.573132 16763 raft_consensus.cc:504] T 9d78abde8d8d43a1846793bc1af40fa5 P 105794c45c6b461f826ca2ff3ee487a3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08dc04cdd71a4f44863ac7eee140a55e" member_type: VOTER last_known_addr { host: "127.11.102.3" port: 43869 } } peers { permanent_uuid: "ecaa1724bb774d869b5edc0f9cfa49c3" member_type: VOTER last_known_addr { host: "127.11.102.2" port: 42361 } } peers { permanent_uuid: "105794c45c6b461f826ca2ff3ee487a3" member_type: VOTER last_known_addr { host: "127.11.102.1" port: 44995 } }
I20260620 06:07:07.574209 16763 leader_election.cc:288] T 9d78abde8d8d43a1846793bc1af40fa5 P 105794c45c6b461f826ca2ff3ee487a3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 08dc04cdd71a4f44863ac7eee140a55e (127.11.102.3:43869), ecaa1724bb774d869b5edc0f9cfa49c3 (127.11.102.2:42361)
I20260620 06:07:07.576442 16752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:07:07.574539 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.102.1:36589 (local address 127.11.102.3:43869)
0620 06:07:07.574681 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0620 06:07:07.574686 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 06:07:07.574800 (+ 114us) server_negotiation.cc:407] Connection header received
0620 06:07:07.574911 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:07:07.574916 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:07:07.574958 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:07:07.575041 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:07:07.576014 (+ 973us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:07:07.576021 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:07:07.576027 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 06:07:07.576041 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:07:07.576068 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 06:07:07.576080 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:07:07.576168 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0620 06:07:07.576260 (+ 92us) server_negotiation.cc:299] Negotiation successful
0620 06:07:07.576304 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260620 06:07:07.576457 16766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:07:07.574403 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.102.3:43869 (local address 127.11.102.1:36589)
0620 06:07:07.574686 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0620 06:07:07.574702 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 06:07:07.574814 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:07:07.575074 (+ 260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:07:07.575077 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:07:07.575096 (+ 19us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:07:07.575857 (+ 761us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 06:07:07.575858 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:07:07.575868 (+ 10us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 06:07:07.575892 (+ 24us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:07:07.576164 (+ 272us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:07:07.576167 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:07:07.576170 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 06:07:07.576262 (+ 92us) client_negotiation.cc:240] Negotiation successful
0620 06:07:07.576300 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":214,"thread_start_us":86,"threads_started":1}
I20260620 06:07:07.576508 16767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:07:07.574816 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.102.2:42361 (local address 127.11.102.1:57581)
0620 06:07:07.575057 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0620 06:07:07.575069 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 06:07:07.575158 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:07:07.575426 (+ 268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:07:07.575430 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:07:07.575445 (+ 15us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:07:07.575901 (+ 456us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 06:07:07.575905 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:07:07.575920 (+ 15us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 06:07:07.575980 (+ 60us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:07:07.576259 (+ 279us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:07:07.576262 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:07:07.576266 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 06:07:07.576343 (+ 77us) client_negotiation.cc:240] Negotiation successful
0620 06:07:07.576374 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":172,"thread_start_us":65,"threads_started":1}
I20260620 06:07:07.576524 16754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:07:07.574924 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.102.1:57581 (local address 127.11.102.2:42361)
0620 06:07:07.575071 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0620 06:07:07.575075 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 06:07:07.575120 (+ 45us) server_negotiation.cc:407] Connection header received
0620 06:07:07.575258 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:07:07.575264 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:07:07.575308 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:07:07.575382 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:07:07.576104 (+ 722us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:07:07.576107 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:07:07.576109 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 06:07:07.576140 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:07:07.576163 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 06:07:07.576173 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:07:07.576263 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0620 06:07:07.576360 (+ 97us) server_negotiation.cc:299] Negotiation successful
0620 06:07:07.576396 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260620 06:07:07.577049 16694 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d78abde8d8d43a1846793bc1af40fa5" candidate_uuid: "105794c45c6b461f826ca2ff3ee487a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "08dc04cdd71a4f44863ac7eee140a55e" is_pre_election: true
I20260620 06:07:07.577049 16563 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d78abde8d8d43a1846793bc1af40fa5" candidate_uuid: "105794c45c6b461f826ca2ff3ee487a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ecaa1724bb774d869b5edc0f9cfa49c3" is_pre_election: true
I20260620 06:07:07.577350 16694 raft_consensus.cc:2454] T 9d78abde8d8d43a1846793bc1af40fa5 P 08dc04cdd71a4f44863ac7eee140a55e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 105794c45c6b461f826ca2ff3ee487a3 in term 0.
I20260620 06:07:07.577350 16563 raft_consensus.cc:2454] T 9d78abde8d8d43a1846793bc1af40fa5 P ecaa1724bb774d869b5edc0f9cfa49c3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 105794c45c6b461f826ca2ff3ee487a3 in term 0.
I20260620 06:07:07.577811 16368 leader_election.cc:302] T 9d78abde8d8d43a1846793bc1af40fa5 P 105794c45c6b461f826ca2ff3ee487a3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 105794c45c6b461f826ca2ff3ee487a3, ecaa1724bb774d869b5edc0f9cfa49c3; no voters:
I20260620 06:07:07.578083 16763 raft_consensus.cc:2781] T 9d78abde8d8d43a1846793bc1af40fa5 P 105794c45c6b461f826ca2ff3ee487a3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 06:07:07.578171 16763 raft_consensus.cc:482] T 9d78abde8d8d43a1846793bc1af40fa5 P 105794c45c6b461f826ca2ff3ee487a3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 06:07:07.578215 16763 raft_consensus.cc:3037] T 9d78abde8d8d43a1846793bc1af40fa5 P 105794c45c6b461f826ca2ff3ee487a3 [term 0 FOLLOWER]: Advancing to term 1
I20260620 06:07:07.579221 16763 raft_consensus.cc:504] T 9d78abde8d8d43a1846793bc1af40fa5 P 105794c45c6b461f826ca2ff3ee487a3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08dc04cdd71a4f44863ac7eee140a55e" member_type: VOTER last_known_addr { host: "127.11.102.3" port: 43869 } } peers { permanent_uuid: "ecaa1724bb774d869b5edc0f9cfa49c3" member_type: VOTER last_known_addr { host: "127.11.102.2" port: 42361 } } peers { permanent_uuid: "105794c45c6b461f826ca2ff3ee487a3" member_type: VOTER last_known_addr { host: "127.11.102.1" port: 44995 } }
I20260620 06:07:07.579538 16763 leader_election.cc:288] T 9d78abde8d8d43a1846793bc1af40fa5 P 105794c45c6b461f826ca2ff3ee487a3 [CANDIDATE]: Term 1 election: Requested vote from peers 08dc04cdd71a4f44863ac7eee140a55e (127.11.102.3:43869), ecaa1724bb774d869b5edc0f9cfa49c3 (127.11.102.2:42361)
I20260620 06:07:07.579859 16694 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d78abde8d8d43a1846793bc1af40fa5" candidate_uuid: "105794c45c6b461f826ca2ff3ee487a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "08dc04cdd71a4f44863ac7eee140a55e"
I20260620 06:07:07.579967 16694 raft_consensus.cc:3037] T 9d78abde8d8d43a1846793bc1af40fa5 P 08dc04cdd71a4f44863ac7eee140a55e [term 0 FOLLOWER]: Advancing to term 1
I20260620 06:07:07.579993 16563 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d78abde8d8d43a1846793bc1af40fa5" candidate_uuid: "105794c45c6b461f826ca2ff3ee487a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ecaa1724bb774d869b5edc0f9cfa49c3"
I20260620 06:07:07.580093 16563 raft_consensus.cc:3037] T 9d78abde8d8d43a1846793bc1af40fa5 P ecaa1724bb774d869b5edc0f9cfa49c3 [term 0 FOLLOWER]: Advancing to term 1
I20260620 06:07:07.580909 16694 raft_consensus.cc:2454] T 9d78abde8d8d43a1846793bc1af40fa5 P 08dc04cdd71a4f44863ac7eee140a55e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 105794c45c6b461f826ca2ff3ee487a3 in term 1.
I20260620 06:07:07.581194 16368 leader_election.cc:302] T 9d78abde8d8d43a1846793bc1af40fa5 P 105794c45c6b461f826ca2ff3ee487a3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 08dc04cdd71a4f44863ac7eee140a55e, 105794c45c6b461f826ca2ff3ee487a3; no voters:
I20260620 06:07:07.581200 16563 raft_consensus.cc:2454] T 9d78abde8d8d43a1846793bc1af40fa5 P ecaa1724bb774d869b5edc0f9cfa49c3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 105794c45c6b461f826ca2ff3ee487a3 in term 1.
I20260620 06:07:07.581390 16763 raft_consensus.cc:2781] T 9d78abde8d8d43a1846793bc1af40fa5 P 105794c45c6b461f826ca2ff3ee487a3 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 06:07:07.581620 16763 raft_consensus.cc:686] T 9d78abde8d8d43a1846793bc1af40fa5 P 105794c45c6b461f826ca2ff3ee487a3 [term 1 LEADER]: Becoming Leader. State: Replica: 105794c45c6b461f826ca2ff3ee487a3, State: Running, Role: LEADER
I20260620 06:07:07.581969 16763 consensus_queue.cc:237] T 9d78abde8d8d43a1846793bc1af40fa5 P 105794c45c6b461f826ca2ff3ee487a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08dc04cdd71a4f44863ac7eee140a55e" member_type: VOTER last_known_addr { host: "127.11.102.3" port: 43869 } } peers { permanent_uuid: "ecaa1724bb774d869b5edc0f9cfa49c3" member_type: VOTER last_known_addr { host: "127.11.102.2" port: 42361 } } peers { permanent_uuid: "105794c45c6b461f826ca2ff3ee487a3" member_type: VOTER last_known_addr { host: "127.11.102.1" port: 44995 } }
I20260620 06:07:07.585000 16289 catalog_manager.cc:5495] T 9d78abde8d8d43a1846793bc1af40fa5 P 105794c45c6b461f826ca2ff3ee487a3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 105794c45c6b461f826ca2ff3ee487a3 (127.11.102.1). New cstate: current_term: 1 leader_uuid: "105794c45c6b461f826ca2ff3ee487a3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08dc04cdd71a4f44863ac7eee140a55e" member_type: VOTER last_known_addr { host: "127.11.102.3" port: 43869 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ecaa1724bb774d869b5edc0f9cfa49c3" member_type: VOTER last_known_addr { host: "127.11.102.2" port: 42361 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "105794c45c6b461f826ca2ff3ee487a3" member_type: VOTER last_known_addr { host: "127.11.102.1" port: 44995 } health_report { overall_health: HEALTHY } } }
I20260620 06:07:07.612324 16756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:07:07.610591 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52818 (local address 127.11.102.1:44995)
0620 06:07:07.610751 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0620 06:07:07.610755 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 06:07:07.610769 (+ 14us) server_negotiation.cc:407] Connection header received
0620 06:07:07.611019 (+ 250us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:07:07.611022 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:07:07.611084 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:07:07.611179 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:07:07.611936 (+ 757us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:07:07.611940 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:07:07.611943 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 06:07:07.611954 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:07:07.611977 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 06:07:07.611988 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:07:07.612066 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0620 06:07:07.612133 (+ 67us) server_negotiation.cc:299] Negotiation successful
0620 06:07:07.612172 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260620 06:07:07.618199 16563 raft_consensus.cc:1261] T 9d78abde8d8d43a1846793bc1af40fa5 P ecaa1724bb774d869b5edc0f9cfa49c3 [term 1 FOLLOWER]: Refusing update from remote peer 105794c45c6b461f826ca2ff3ee487a3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 06:07:07.618213 16694 raft_consensus.cc:1261] T 9d78abde8d8d43a1846793bc1af40fa5 P 08dc04cdd71a4f44863ac7eee140a55e [term 1 FOLLOWER]: Refusing update from remote peer 105794c45c6b461f826ca2ff3ee487a3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 06:07:07.618956 16763 consensus_queue.cc:1035] T 9d78abde8d8d43a1846793bc1af40fa5 P 105794c45c6b461f826ca2ff3ee487a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "08dc04cdd71a4f44863ac7eee140a55e" member_type: VOTER last_known_addr { host: "127.11.102.3" port: 43869 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 06:07:07.619143 16768 consensus_queue.cc:1035] T 9d78abde8d8d43a1846793bc1af40fa5 P 105794c45c6b461f826ca2ff3ee487a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ecaa1724bb774d869b5edc0f9cfa49c3" member_type: VOTER last_known_addr { host: "127.11.102.2" port: 42361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 06:07:07.626803 16771 mvcc.cc:205] Tried to move back new op lower bound from 7298808330717163520 to 7298808330578423808. Current Snapshot: MvccSnapshot[applied={T|T < 7298808330717163520}]
I20260620 06:07:07.627653 16773 mvcc.cc:205] Tried to move back new op lower bound from 7298808330717163520 to 7298808330578423808. Current Snapshot: MvccSnapshot[applied={T|T < 7298808330717163520}]
I20260620 06:07:07.634789 16290 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:50938:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 06:07:07.634953 16290 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:50938:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 06:07:07.637252 16290 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2a7b97dea0a64ea1b04418276432191a: Sending DeleteTablet for 3 replicas of tablet 9d78abde8d8d43a1846793bc1af40fa5
I20260620 06:07:07.637917 16674 tablet_service.cc:1514] Processing DeleteTablet for tablet 9d78abde8d8d43a1846793bc1af40fa5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 06:07:07 UTC) from {username='slave'} at 127.0.0.1:34270
I20260620 06:07:07.638046 16543 tablet_service.cc:1514] Processing DeleteTablet for tablet 9d78abde8d8d43a1846793bc1af40fa5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 06:07:07 UTC) from {username='slave'} at 127.0.0.1:41146
I20260620 06:07:07.638047 16412 tablet_service.cc:1514] Processing DeleteTablet for tablet 9d78abde8d8d43a1846793bc1af40fa5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 06:07:07 UTC) from {username='slave'} at 127.0.0.1:52814
I20260620 06:07:07.638455 16782 tablet_replica.cc:330] T 9d78abde8d8d43a1846793bc1af40fa5 P ecaa1724bb774d869b5edc0f9cfa49c3: stopping tablet replica
I20260620 06:07:07.638664 16783 tablet_replica.cc:330] T 9d78abde8d8d43a1846793bc1af40fa5 P 105794c45c6b461f826ca2ff3ee487a3: stopping tablet replica
I20260620 06:07:07.638675 11672 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9HMp43/build/debug/bin/kudu with pid 16350
I20260620 06:07:07.639005 16783 raft_consensus.cc:2229] T 9d78abde8d8d43a1846793bc1af40fa5 P 105794c45c6b461f826ca2ff3ee487a3 [term 1 LEADER]: Raft consensus shutting down.
I20260620 06:07:07.639463 16781 tablet_replica.cc:330] T 9d78abde8d8d43a1846793bc1af40fa5 P 08dc04cdd71a4f44863ac7eee140a55e: stopping tablet replica
I20260620 06:07:07.639842 16781 raft_consensus.cc:2229] T 9d78abde8d8d43a1846793bc1af40fa5 P 08dc04cdd71a4f44863ac7eee140a55e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 06:07:07.640190 16781 raft_consensus.cc:2258] T 9d78abde8d8d43a1846793bc1af40fa5 P 08dc04cdd71a4f44863ac7eee140a55e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 06:07:07.638773 16782 raft_consensus.cc:2229] T 9d78abde8d8d43a1846793bc1af40fa5 P ecaa1724bb774d869b5edc0f9cfa49c3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 06:07:07.641161 16782 raft_consensus.cc:2258] T 9d78abde8d8d43a1846793bc1af40fa5 P ecaa1724bb774d869b5edc0f9cfa49c3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 06:07:07.641991 16781 ts_tablet_manager.cc:1905] T 9d78abde8d8d43a1846793bc1af40fa5 P 08dc04cdd71a4f44863ac7eee140a55e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 06:07:07.642393 16782 ts_tablet_manager.cc:1905] T 9d78abde8d8d43a1846793bc1af40fa5 P ecaa1724bb774d869b5edc0f9cfa49c3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 06:07:07.644559 16781 ts_tablet_manager.cc:1918] T 9d78abde8d8d43a1846793bc1af40fa5 P 08dc04cdd71a4f44863ac7eee140a55e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 06:07:07.644644 16781 log.cc:1196] T 9d78abde8d8d43a1846793bc1af40fa5 P 08dc04cdd71a4f44863ac7eee140a55e: Deleting WAL directory at /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781935603120653-11672-0/minicluster-data/ts-2/wal/wals/9d78abde8d8d43a1846793bc1af40fa5
I20260620 06:07:07.644950 16781 ts_tablet_manager.cc:1939] T 9d78abde8d8d43a1846793bc1af40fa5 P 08dc04cdd71a4f44863ac7eee140a55e: Deleting consensus metadata
I20260620 06:07:07.645443 16782 ts_tablet_manager.cc:1918] T 9d78abde8d8d43a1846793bc1af40fa5 P ecaa1724bb774d869b5edc0f9cfa49c3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 06:07:07.645534 16782 log.cc:1196] T 9d78abde8d8d43a1846793bc1af40fa5 P ecaa1724bb774d869b5edc0f9cfa49c3: Deleting WAL directory at /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781935603120653-11672-0/minicluster-data/ts-1/wal/wals/9d78abde8d8d43a1846793bc1af40fa5
I20260620 06:07:07.645771 11672 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9HMp43/build/debug/bin/kudu with pid 16481
I20260620 06:07:07.645874 16782 ts_tablet_manager.cc:1939] T 9d78abde8d8d43a1846793bc1af40fa5 P ecaa1724bb774d869b5edc0f9cfa49c3: Deleting consensus metadata
W20260620 06:07:07.647414 16277 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.11.102.1:44995 (error 108)
I20260620 06:07:07.648028 16277 catalog_manager.cc:4841] TS 08dc04cdd71a4f44863ac7eee140a55e (127.11.102.3:43869): tablet 9d78abde8d8d43a1846793bc1af40fa5 (table test-table [id=1225e00e238d4c6c979d3f663ddaa298]) successfully deleted
I20260620 06:07:07.648782 16753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:07:07.648408 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.102.1:44995 (local address 127.0.0.1:52826)
0620 06:07:07.648592 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0620 06:07:07.648682 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.102.1:44995: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":90}
W20260620 06:07:07.648941 16277 catalog_manager.cc:4584] TS 105794c45c6b461f826ca2ff3ee487a3 (127.11.102.1:44995): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 9d78abde8d8d43a1846793bc1af40fa5: Network error: Client connection negotiation failed: client connection to 127.11.102.1:44995: connect: Connection refused (error 111)
I20260620 06:07:07.653084 11672 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9HMp43/build/debug/bin/kudu with pid 16612
I20260620 06:07:07.653935 16753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:07:07.653272 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.102.2:42361 (local address 127.0.0.1:41154)
0620 06:07:07.653835 (+ 563us) negotiation.cc:107] Waiting for socket to connect
0620 06:07:07.653878 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.102.2:42361: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":499}
I20260620 06:07:07.658823 11672 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9HMp43/build/debug/bin/kudu with pid 16259
2026-06-20T06:07:07Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (1010 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260620 06:07:07.680276 11672 mini_oidc.cc:152] Starting JWKS server
I20260620 06:07:07.681828 11672 webserver.cc:460] Webserver started at https://127.0.0.1:44693/ using document root <none> and password file <none>
I20260620 06:07:07.681998 11672 mini_oidc.cc:189] Starting OIDC Discovery server
I20260620 06:07:07.682276 11672 webserver.cc:460] Webserver started at http://127.0.0.1:45709/ using document root <none> and password file <none>
2026-06-20T06:07:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T06:07:07Z Disabled control of system clock
I20260620 06:07:07.690907 11672 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9HMp43/build/debug/bin/kudu
/tmp/dist-test-task9HMp43/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781935603120653-11672-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781935603120653-11672-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781935603120653-11672-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781935603120653-11672-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.102.62:43311
--webserver_interface=127.11.102.62
--webserver_port=0
--builtin_ntp_servers=127.11.102.20:37979
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.102.62:43311
--enable_jwt_token_auth=true
--jwks_url=https://localhost:44693/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781935603120653-11672-0/testchainca.cert with env {}
W20260620 06:07:07.796097 16790 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:07:07.796327 16790 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:07:07.796365 16790 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:07:07.800241 16790 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 06:07:07.800290 16790 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:07:07.800305 16790 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:07:07.800323 16790 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 06:07:07.800382 16790 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260620 06:07:07.800403 16790 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260620 06:07:07.800431 16790 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:44693/jwks.json
I20260620 06:07:07.805645 16790 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.102.20:37979
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781935603120653-11672-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781935603120653-11672-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.102.62:43311
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.102.62:43311
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:44693/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781935603120653-11672-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.102.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781935603120653-11672-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16790
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781935603120653-11672-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 05:28:48 UTC on ccycloud.quasar-fixhms.root.comops.site
build id 4457020
I20260620 06:07:07.806555 16790 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:07:07.807492 16790 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 06:07:07.815497 16796 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:07:07.815459 16795 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:07:07.815572 16798 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:07:07.815474 16790 server_base.cc:1034] running on GCE node
I20260620 06:07:07.816279 16790 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:07:07.817369 16790 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:07:07.818552 16790 hybrid_clock.cc:648] HybridClock initialized: now 1781935627818538 us; error 34 us; skew 500 ppm
I20260620 06:07:07.820369 16790 webserver.cc:460] Webserver started at http://127.11.102.62:33951/ using document root <none> and password file <none>
I20260620 06:07:07.820940 16790 fs_manager.cc:359] Metadata directory not provided
I20260620 06:07:07.821000 16790 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:07:07.821206 16790 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 06:07:07.823279 16790 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781935603120653-11672-0/minicluster-data/master-0/data/instance:
uuid: "b3d61a90280a43b393e531fa8ac84e3d"
format_stamp: "Formatted at 2026-06-20 06:07:07 on dist-test-slave-9hm5"
server_key: "444a86a1ca703104cc1f94175b6cb104"
server_key_iv: "f87e9fa9cbf6a682e66a0cdd748008a3"
server_key_version: "encryptionkey@0"
I20260620 06:07:07.823877 16790 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781935603120653-11672-0/minicluster-data/master-0/wal/instance:
uuid: "b3d61a90280a43b393e531fa8ac84e3d"
format_stamp: "Formatted at 2026-06-20 06:07:07 on dist-test-slave-9hm5"
server_key: "444a86a1ca703104cc1f94175b6cb104"
server_key_iv: "f87e9fa9cbf6a682e66a0cdd748008a3"
server_key_version: "encryptionkey@0"
I20260620 06:07:07.827553 16790 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260620 06:07:07.830073 16804 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:07:07.831367 16790 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260620 06:07:07.831518 16790 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781935603120653-11672-0/minicluster-data/master-0/data,/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781935603120653-11672-0/minicluster-data/master-0/wal
uuid: "b3d61a90280a43b393e531fa8ac84e3d"
format_stamp: "Formatted at 2026-06-20 06:07:07 on dist-test-slave-9hm5"
server_key: "444a86a1ca703104cc1f94175b6cb104"
server_key_iv: "f87e9fa9cbf6a682e66a0cdd748008a3"
server_key_version: "encryptionkey@0"
I20260620 06:07:07.831658 16790 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781935603120653-11672-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781935603120653-11672-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781935603120653-11672-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:07:07.841557 16790 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:07:07.842181 16790 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:07:07.842377 16790 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:07:07.849274 16790 rpc_server.cc:307] RPC server started. Bound to: 127.11.102.62:43311
I20260620 06:07:07.849294 16856 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.102.62:43311 every 8 connection(s)
I20260620 06:07:07.850214 16790 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781935603120653-11672-0/minicluster-data/master-0/data/info.pb
I20260620 06:07:07.853338 16857 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 06:07:07.856186 11672 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9HMp43/build/debug/bin/kudu as pid 16790
I20260620 06:07:07.856312 11672 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781935603120653-11672-0/minicluster-data/master-0/wal/instance
I20260620 06:07:07.856623 11672 external_mini_cluster.cc:1428] Setting key 6e60ac8be05a1b2ee635be3d71469b2e
I20260620 06:07:07.858954 16857 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b3d61a90280a43b393e531fa8ac84e3d: Bootstrap starting.
I20260620 06:07:07.861313 16857 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b3d61a90280a43b393e531fa8ac84e3d: Neither blocks nor log segments found. Creating new log.
I20260620 06:07:07.862169 16857 log.cc:824] T 00000000000000000000000000000000 P b3d61a90280a43b393e531fa8ac84e3d: Log is configured to *not* fsync() on all Append() calls
I20260620 06:07:07.864475 16857 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b3d61a90280a43b393e531fa8ac84e3d: No bootstrap required, opened a new log
I20260620 06:07:07.866132 16860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:07:07.857888 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38666 (local address 127.11.102.62:43311)
0620 06:07:07.858468 (+ 580us) server_negotiation.cc:206] Beginning negotiation
0620 06:07:07.858483 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 06:07:07.858518 (+ 35us) server_negotiation.cc:407] Connection header received
0620 06:07:07.859190 (+ 672us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:07:07.859208 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:07:07.859589 (+ 381us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:07:07.859937 (+ 348us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:07:07.860283 (+ 346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:07:07.861214 (+ 931us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:07:07.862442 (+ 1228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:07:07.863090 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:07:07.863259 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:07:07.864072 (+ 813us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:07:07.864078 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:07:07.864094 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 06:07:07.864602 (+ 508us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:07:07.864674 (+ 72us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 06:07:07.864703 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:07:07.864810 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0620 06:07:07.864921 (+ 111us) server_negotiation.cc:299] Negotiation successful
0620 06:07:07.865198 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":245,"thread_start_us":133,"threads_started":1}
I20260620 06:07:07.868954 11672 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 06:07:07.869220 16857 raft_consensus.cc:348] T 00000000000000000000000000000000 P b3d61a90280a43b393e531fa8ac84e3d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b3d61a90280a43b393e531fa8ac84e3d" member_type: VOTER last_known_addr { host: "127.11.102.62" port: 43311 } }
I20260620 06:07:07.869431 16857 raft_consensus.cc:374] T 00000000000000000000000000000000 P b3d61a90280a43b393e531fa8ac84e3d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 06:07:07.869517 16857 raft_consensus.cc:729] T 00000000000000000000000000000000 P b3d61a90280a43b393e531fa8ac84e3d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b3d61a90280a43b393e531fa8ac84e3d, State: Initialized, Role: FOLLOWER
I20260620 06:07:07.869992 16857 consensus_queue.cc:260] T 00000000000000000000000000000000 P b3d61a90280a43b393e531fa8ac84e3d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b3d61a90280a43b393e531fa8ac84e3d" member_type: VOTER last_known_addr { host: "127.11.102.62" port: 43311 } }
I20260620 06:07:07.870153 16857 raft_consensus.cc:388] T 00000000000000000000000000000000 P b3d61a90280a43b393e531fa8ac84e3d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 06:07:07.870229 16857 raft_consensus.cc:482] T 00000000000000000000000000000000 P b3d61a90280a43b393e531fa8ac84e3d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 06:07:07.870332 16857 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b3d61a90280a43b393e531fa8ac84e3d [term 0 FOLLOWER]: Advancing to term 1
I20260620 06:07:07.871340 16857 raft_consensus.cc:504] T 00000000000000000000000000000000 P b3d61a90280a43b393e531fa8ac84e3d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b3d61a90280a43b393e531fa8ac84e3d" member_type: VOTER last_known_addr { host: "127.11.102.62" port: 43311 } }
I20260620 06:07:07.871771 16857 leader_election.cc:302] T 00000000000000000000000000000000 P b3d61a90280a43b393e531fa8ac84e3d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b3d61a90280a43b393e531fa8ac84e3d; no voters:
I20260620 06:07:07.872116 16857 leader_election.cc:288] T 00000000000000000000000000000000 P b3d61a90280a43b393e531fa8ac84e3d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 06:07:07.872200 16862 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b3d61a90280a43b393e531fa8ac84e3d [term 1 FOLLOWER]: Leader election won for term 1
I20260620 06:07:07.872483 16862 raft_consensus.cc:686] T 00000000000000000000000000000000 P b3d61a90280a43b393e531fa8ac84e3d [term 1 LEADER]: Becoming Leader. State: Replica: b3d61a90280a43b393e531fa8ac84e3d, State: Running, Role: LEADER
I20260620 06:07:07.872786 16862 consensus_queue.cc:237] T 00000000000000000000000000000000 P b3d61a90280a43b393e531fa8ac84e3d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b3d61a90280a43b393e531fa8ac84e3d" member_type: VOTER last_known_addr { host: "127.11.102.62" port: 43311 } }
I20260620 06:07:07.873379 16857 sys_catalog.cc:564] T 00000000000000000000000000000000 P b3d61a90280a43b393e531fa8ac84e3d [sys.catalog]: configured and running, proceeding with master startup.
I20260620 06:07:07.874401 16863 sys_catalog.cc:455] T 00000000000000000000000000000000 P b3d61a90280a43b393e531fa8ac84e3d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b3d61a90280a43b393e531fa8ac84e3d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b3d61a90280a43b393e531fa8ac84e3d" member_type: VOTER last_known_addr { host: "127.11.102.62" port: 43311 } } }
I20260620 06:07:07.874534 16863 sys_catalog.cc:458] T 00000000000000000000000000000000 P b3d61a90280a43b393e531fa8ac84e3d [sys.catalog]: This master's current role is: LEADER
I20260620 06:07:07.874788 16864 sys_catalog.cc:455] T 00000000000000000000000000000000 P b3d61a90280a43b393e531fa8ac84e3d [sys.catalog]: SysCatalogTable state changed. Reason: New leader b3d61a90280a43b393e531fa8ac84e3d. Latest consensus state: current_term: 1 leader_uuid: "b3d61a90280a43b393e531fa8ac84e3d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b3d61a90280a43b393e531fa8ac84e3d" member_type: VOTER last_known_addr { host: "127.11.102.62" port: 43311 } } }
I20260620 06:07:07.874882 16864 sys_catalog.cc:458] T 00000000000000000000000000000000 P b3d61a90280a43b393e531fa8ac84e3d [sys.catalog]: This master's current role is: LEADER
I20260620 06:07:07.874974 16871 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 06:07:07.877669 16871 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 06:07:07.882719 16871 catalog_manager.cc:1347] Generated new cluster ID: e70da58db75d48b09c7ea9ccc24414c7
I20260620 06:07:07.882781 16871 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 06:07:07.891312 16871 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 06:07:07.892589 16871 catalog_manager.cc:1504] Loading token signing keys...
I20260620 06:07:07.904812 16871 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b3d61a90280a43b393e531fa8ac84e3d: Generated new TSK 0
I20260620 06:07:07.905625 16871 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260620 06:07:07.932885 16887 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260620 06:07:07.933002 16887 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260620 06:07:07.933496 16860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:07:07.917151 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38668 (local address 127.11.102.62:43311)
0620 06:07:07.917295 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0620 06:07:07.917299 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 06:07:07.917580 (+ 281us) server_negotiation.cc:407] Connection header received
0620 06:07:07.917769 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:07:07.917772 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:07:07.917816 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:07:07.917897 (+ 81us) server_negotiation.cc:226] Negotiated authn=JWT
0620 06:07:07.918169 (+ 272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:07:07.918680 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:07:07.919891 (+ 1211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:07:07.920108 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:07:07.920184 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:07:07.920523 (+ 339us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0620 06:07:07.933174 (+ 12651us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44693/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0620 06:07:07.933390 (+ 216us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:38668: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44693/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":53}
W20260620 06:07:07.933579 16860 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:38668: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44693/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260620 06:07:07.934928 16886 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 06:07:07.917107 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.102.62:43311 (local address 127.0.0.1:38668)
0620 06:07:07.917476 (+ 369us) negotiation.cc:107] Waiting for socket to connect
0620 06:07:07.917491 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 06:07:07.917617 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:07:07.917929 (+ 312us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:07:07.917932 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:07:07.917940 (+ 8us) client_negotiation.cc:189] Negotiated authn=JWT
0620 06:07:07.918069 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:07:07.918075 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:07:07.918797 (+ 722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:07:07.918800 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:07:07.919785 (+ 985us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:07:07.919795 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:07:07.920217 (+ 422us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:07:07.920221 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:07:07.920310 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:07:07.920354 (+ 44us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0620 06:07:07.933390 (+ 13036us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44693/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0620 06:07:07.933626 (+ 236us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.11.102.62:43311: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44693/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":229,"thread_start_us":133,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.11.102.62:43311: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44693/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260620 06:07:07.935727 11672 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9HMp43/build/debug/bin/kudu with pid 16790
2026-06-20T06:07:07Z chronyd exiting
I20260620 06:07:07.972513 11672 test_util.cc:175] -----------------------------------------------
I20260620 06:07:07.972591 11672 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781935603120653-11672-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (293 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260620 06:07:07.973572 11672 mini_oidc.cc:152] Starting JWKS server
I20260620 06:07:07.974370 11672 webserver.cc:460] Webserver started at https://127.0.0.1:40219/ using document root <none> and password file <none>
I20260620 06:07:07.974448 11672 mini_oidc.cc:189] Starting OIDC Discovery server
I20260620 06:07:07.974710 11672 webserver.cc:460] Webserver started at http://127.0.0.1:33421/ using document root <none> and password file <none>
2026-06-20T06:07:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T06:07:07Z Disabled control of system clock
I20260620 06:07:07.982967 11672 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9HMp43/build/debug/bin/kudu
/tmp/dist-test-task9HMp43/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781935603120653-11672-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781935603120653-11672-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781935603120653-11672-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781935603120653-11672-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.102.62:42895
--webserver_interface=127.11.102.62
--webserver_port=0
--builtin_ntp_servers=127.11.102.20:33215
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.102.62:42895
--enable_jwt_token_auth=true
--jwks_url=https://localhost:40219/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781935603120653-11672-0/testchainca.cert with env {}
W20260620 06:07:08.087215 16893 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:07:08.087482 16893 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:07:08.087553 16893 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:07:08.091457 16893 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 06:07:08.091542 16893 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:07:08.091571 16893 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:07:08.091631 16893 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 06:07:08.091686 16893 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260620 06:07:08.091727 16893 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260620 06:07:08.091758 16893 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:40219/jwks.json
I20260620 06:07:08.097023 16893 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.102.20:33215
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781935603120653-11672-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781935603120653-11672-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.102.62:42895
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.102.62:42895
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:40219/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781935603120653-11672-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.102.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781935603120653-11672-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16893
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781935603120653-11672-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 05:28:48 UTC on ccycloud.quasar-fixhms.root.comops.site
build id 4457020
I20260620 06:07:08.097990 16893 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:07:08.098906 16893 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 06:07:08.106945 16898 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:07:08.106951 16899 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:07:08.107069 16901 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:07:08.107221 16893 server_base.cc:1034] running on GCE node
I20260620 06:07:08.107864 16893 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:07:08.108826 16893 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:07:08.110013 16893 hybrid_clock.cc:648] HybridClock initialized: now 1781935628110003 us; error 25 us; skew 500 ppm
I20260620 06:07:08.111826 16893 webserver.cc:460] Webserver started at http://127.11.102.62:35027/ using document root <none> and password file <none>
I20260620 06:07:08.112353 16893 fs_manager.cc:359] Metadata directory not provided
I20260620 06:07:08.112455 16893 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:07:08.112666 16893 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 06:07:08.114566 16893 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781935603120653-11672-0/minicluster-data/master-0/data/instance:
uuid: "86a453012fb4455f8e3c38a220f8a6d4"
format_stamp: "Formatted at 2026-06-20 06:07:08 on dist-test-slave-9hm5"
server_key: "0e0eda8db63f59e4aa13eb4a23f7c737"
server_key_iv: "4a9d1b9bb8a91073ad6f9b8c69a58d50"
server_key_version: "encryptionkey@0"
I20260620 06:07:08.115095 16893 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781935603120653-11672-0/minicluster-data/master-0/wal/instance:
uuid: "86a453012fb4455f8e3c38a220f8a6d4"
format_stamp: "Formatted at 2026-06-20 06:07:08 on dist-test-slave-9hm5"
server_key: "0e0eda8db63f59e4aa13eb4a23f7c737"
server_key_iv: "4a9d1b9bb8a91073ad6f9b8c69a58d50"
server_key_version: "encryptionkey@0"
I20260620 06:07:08.118572 16893 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.003s
I20260620 06:07:08.120837 16907 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:07:08.122004 16893 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260620 06:07:08.122136 16893 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781935603120653-11672-0/minicluster-data/master-0/data,/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781935603120653-11672-0/minicluster-data/master-0/wal
uuid: "86a453012fb4455f8e3c38a220f8a6d4"
format_stamp: "Formatted at 2026-06-20 06:07:08 on dist-test-slave-9hm5"
server_key: "0e0eda8db63f59e4aa13eb4a23f7c737"
server_key_iv: "4a9d1b9bb8a91073ad6f9b8c69a58d50"
server_key_version: "encryptionkey@0"
I20260620 06:07:08.122241 16893 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781935603120653-11672-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781935603120653-11672-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781935603120653-11672-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:07:08.146024 16893 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:07:08.146680 16893 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:07:08.146871 16893 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:07:08.153745 16893 rpc_server.cc:307] RPC server started. Bound to: 127.11.102.62:42895
I20260620 06:07:08.153753 16959 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.102.62:42895 every 8 connection(s)
I20260620 06:07:08.154636 16893 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781935603120653-11672-0/minicluster-data/master-0/data/info.pb
I20260620 06:07:08.157670 11672 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9HMp43/build/debug/bin/kudu as pid 16893
I20260620 06:07:08.157709 16960 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 06:07:08.157758 11672 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781935603120653-11672-0/minicluster-data/master-0/wal/instance
I20260620 06:07:08.158078 11672 external_mini_cluster.cc:1428] Setting key 2424f0a79c1573ce8039c16009dded1d
I20260620 06:07:08.164587 16960 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 86a453012fb4455f8e3c38a220f8a6d4: Bootstrap starting.
I20260620 06:07:08.167096 16960 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 86a453012fb4455f8e3c38a220f8a6d4: Neither blocks nor log segments found. Creating new log.
I20260620 06:07:08.167937 16960 log.cc:824] T 00000000000000000000000000000000 P 86a453012fb4455f8e3c38a220f8a6d4: Log is configured to *not* fsync() on all Append() calls
I20260620 06:07:08.168234 16963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:07:08.159406 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40728 (local address 127.11.102.62:42895)
0620 06:07:08.160083 (+ 677us) server_negotiation.cc:206] Beginning negotiation
0620 06:07:08.160102 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0620 06:07:08.160145 (+ 43us) server_negotiation.cc:407] Connection header received
0620 06:07:08.160917 (+ 772us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:07:08.160948 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:07:08.161440 (+ 492us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:07:08.161798 (+ 358us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:07:08.162226 (+ 428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:07:08.163713 (+ 1487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:07:08.164597 (+ 884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:07:08.165255 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:07:08.165405 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:07:08.166245 (+ 840us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:07:08.166251 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:07:08.166265 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 06:07:08.166722 (+ 457us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:07:08.166789 (+ 67us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 06:07:08.166814 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:07:08.166910 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0620 06:07:08.167075 (+ 165us) server_negotiation.cc:299] Negotiation successful
0620 06:07:08.167348 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":275,"thread_start_us":150,"threads_started":1}
I20260620 06:07:08.169970 16960 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 86a453012fb4455f8e3c38a220f8a6d4: No bootstrap required, opened a new log
I20260620 06:07:08.171032 11672 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 06:07:08.173678 16960 raft_consensus.cc:348] T 00000000000000000000000000000000 P 86a453012fb4455f8e3c38a220f8a6d4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86a453012fb4455f8e3c38a220f8a6d4" member_type: VOTER last_known_addr { host: "127.11.102.62" port: 42895 } }
I20260620 06:07:08.173874 16960 raft_consensus.cc:374] T 00000000000000000000000000000000 P 86a453012fb4455f8e3c38a220f8a6d4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 06:07:08.173949 16960 raft_consensus.cc:729] T 00000000000000000000000000000000 P 86a453012fb4455f8e3c38a220f8a6d4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 86a453012fb4455f8e3c38a220f8a6d4, State: Initialized, Role: FOLLOWER
I20260620 06:07:08.174360 16960 consensus_queue.cc:260] T 00000000000000000000000000000000 P 86a453012fb4455f8e3c38a220f8a6d4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86a453012fb4455f8e3c38a220f8a6d4" member_type: VOTER last_known_addr { host: "127.11.102.62" port: 42895 } }
I20260620 06:07:08.174502 16960 raft_consensus.cc:388] T 00000000000000000000000000000000 P 86a453012fb4455f8e3c38a220f8a6d4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 06:07:08.174584 16960 raft_consensus.cc:482] T 00000000000000000000000000000000 P 86a453012fb4455f8e3c38a220f8a6d4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 06:07:08.174693 16960 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 86a453012fb4455f8e3c38a220f8a6d4 [term 0 FOLLOWER]: Advancing to term 1
I20260620 06:07:08.175477 16960 raft_consensus.cc:504] T 00000000000000000000000000000000 P 86a453012fb4455f8e3c38a220f8a6d4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86a453012fb4455f8e3c38a220f8a6d4" member_type: VOTER last_known_addr { host: "127.11.102.62" port: 42895 } }
I20260620 06:07:08.175827 16960 leader_election.cc:302] T 00000000000000000000000000000000 P 86a453012fb4455f8e3c38a220f8a6d4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 86a453012fb4455f8e3c38a220f8a6d4; no voters:
I20260620 06:07:08.176141 16960 leader_election.cc:288] T 00000000000000000000000000000000 P 86a453012fb4455f8e3c38a220f8a6d4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 06:07:08.176234 16965 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 86a453012fb4455f8e3c38a220f8a6d4 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 06:07:08.176450 16965 raft_consensus.cc:686] T 00000000000000000000000000000000 P 86a453012fb4455f8e3c38a220f8a6d4 [term 1 LEADER]: Becoming Leader. State: Replica: 86a453012fb4455f8e3c38a220f8a6d4, State: Running, Role: LEADER
I20260620 06:07:08.176748 16965 consensus_queue.cc:237] T 00000000000000000000000000000000 P 86a453012fb4455f8e3c38a220f8a6d4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86a453012fb4455f8e3c38a220f8a6d4" member_type: VOTER last_known_addr { host: "127.11.102.62" port: 42895 } }
I20260620 06:07:08.177233 16960 sys_catalog.cc:564] T 00000000000000000000000000000000 P 86a453012fb4455f8e3c38a220f8a6d4 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 06:07:08.178717 16966 sys_catalog.cc:455] T 00000000000000000000000000000000 P 86a453012fb4455f8e3c38a220f8a6d4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "86a453012fb4455f8e3c38a220f8a6d4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86a453012fb4455f8e3c38a220f8a6d4" member_type: VOTER last_known_addr { host: "127.11.102.62" port: 42895 } } }
I20260620 06:07:08.178833 16966 sys_catalog.cc:458] T 00000000000000000000000000000000 P 86a453012fb4455f8e3c38a220f8a6d4 [sys.catalog]: This master's current role is: LEADER
I20260620 06:07:08.179069 16967 sys_catalog.cc:455] T 00000000000000000000000000000000 P 86a453012fb4455f8e3c38a220f8a6d4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 86a453012fb4455f8e3c38a220f8a6d4. Latest consensus state: current_term: 1 leader_uuid: "86a453012fb4455f8e3c38a220f8a6d4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86a453012fb4455f8e3c38a220f8a6d4" member_type: VOTER last_known_addr { host: "127.11.102.62" port: 42895 } } }
I20260620 06:07:08.179148 16967 sys_catalog.cc:458] T 00000000000000000000000000000000 P 86a453012fb4455f8e3c38a220f8a6d4 [sys.catalog]: This master's current role is: LEADER
W20260620 06:07:08.181684 16980 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 86a453012fb4455f8e3c38a220f8a6d4: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 06:07:08.181771 16980 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 06:07:08.181849 16978 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 06:07:08.182497 16978 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 06:07:08.187589 16978 catalog_manager.cc:1347] Generated new cluster ID: ffc90f33f2bb4dc0b92f3c271867169c
I20260620 06:07:08.187667 16978 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 06:07:08.194303 16978 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 06:07:08.195056 16978 catalog_manager.cc:1504] Loading token signing keys...
I20260620 06:07:08.203015 16978 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 86a453012fb4455f8e3c38a220f8a6d4: Generated new TSK 0
I20260620 06:07:08.203625 16978 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260620 06:07:08.237059 16990 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260620 06:07:08.237147 16990 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260620 06:07:08.237699 16963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:07:08.214875 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40738 (local address 127.11.102.62:42895)
0620 06:07:08.215007 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0620 06:07:08.215012 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 06:07:08.215204 (+ 192us) server_negotiation.cc:407] Connection header received
0620 06:07:08.215344 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:07:08.215346 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:07:08.215390 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:07:08.215474 (+ 84us) server_negotiation.cc:226] Negotiated authn=JWT
0620 06:07:08.215710 (+ 236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:07:08.216231 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:07:08.217419 (+ 1188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:07:08.217625 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:07:08.217705 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:07:08.218025 (+ 320us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0620 06:07:08.237373 (+ 19348us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40219/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0620 06:07:08.237593 (+ 220us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:40738: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40219/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":52}
W20260620 06:07:08.237778 16963 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:40738: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40219/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20260620 06:07:08.237898 16989 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 06:07:08.214820 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.102.62:42895 (local address 127.0.0.1:40738)
0620 06:07:08.215113 (+ 293us) negotiation.cc:107] Waiting for socket to connect
0620 06:07:08.215128 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 06:07:08.215237 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:07:08.215491 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:07:08.215494 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:07:08.215502 (+ 8us) client_negotiation.cc:189] Negotiated authn=JWT
0620 06:07:08.215626 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:07:08.215633 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:07:08.216327 (+ 694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:07:08.216330 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:07:08.217303 (+ 973us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:07:08.217312 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:07:08.217742 (+ 430us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:07:08.217746 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:07:08.217832 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:07:08.217857 (+ 25us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0620 06:07:08.237553 (+ 19696us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40219/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0620 06:07:08.237728 (+ 175us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.11.102.62:42895: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40219/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":210,"thread_start_us":113,"threads_started":1}
I20260620 06:07:08.238644 11672 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9HMp43/build/debug/bin/kudu with pid 16893
2026-06-20T06:07:08Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (303 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260620 06:07:08.277058 11672 mini_oidc.cc:152] Starting JWKS server
I20260620 06:07:08.277915 11672 webserver.cc:460] Webserver started at https://127.0.0.1:36385/ using document root <none> and password file <none>
I20260620 06:07:08.278036 11672 mini_oidc.cc:189] Starting OIDC Discovery server
I20260620 06:07:08.278314 11672 webserver.cc:460] Webserver started at http://127.0.0.1:44117/ using document root <none> and password file <none>
2026-06-20T06:07:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T06:07:08Z Disabled control of system clock
I20260620 06:07:08.286870 11672 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9HMp43/build/debug/bin/kudu
/tmp/dist-test-task9HMp43/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781935603120653-11672-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781935603120653-11672-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781935603120653-11672-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781935603120653-11672-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.102.62:44201
--webserver_interface=127.11.102.62
--webserver_port=0
--builtin_ntp_servers=127.11.102.20:33231
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.102.62:44201
--enable_jwt_token_auth=true
--jwks_url=https://localhost:36385/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260620 06:07:08.391461 16996 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:07:08.391724 16996 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:07:08.391808 16996 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:07:08.395738 16996 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 06:07:08.395790 16996 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:07:08.395805 16996 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:07:08.395821 16996 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 06:07:08.395836 16996 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260620 06:07:08.395884 16996 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260620 06:07:08.395915 16996 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:36385/jwks.json
I20260620 06:07:08.400502 16996 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.102.20:33231
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781935603120653-11672-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781935603120653-11672-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.102.62:44201
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.102.62:44201
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:36385/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781935603120653-11672-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.102.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16996
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781935603120653-11672-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 05:28:48 UTC on ccycloud.quasar-fixhms.root.comops.site
build id 4457020
I20260620 06:07:08.401475 16996 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:07:08.402757 16996 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 06:07:08.410786 17004 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:07:08.410782 17002 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:07:08.410796 17001 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:07:08.410998 16996 server_base.cc:1034] running on GCE node
I20260620 06:07:08.411576 16996 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:07:08.412642 16996 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:07:08.413828 16996 hybrid_clock.cc:648] HybridClock initialized: now 1781935628413793 us; error 52 us; skew 500 ppm
I20260620 06:07:08.415555 16996 webserver.cc:460] Webserver started at http://127.11.102.62:41365/ using document root <none> and password file <none>
I20260620 06:07:08.416116 16996 fs_manager.cc:359] Metadata directory not provided
I20260620 06:07:08.416208 16996 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:07:08.416429 16996 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 06:07:08.418313 16996 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781935603120653-11672-0/minicluster-data/master-0/data/instance:
uuid: "9aba9fb589014062ad721ff283c384e0"
format_stamp: "Formatted at 2026-06-20 06:07:08 on dist-test-slave-9hm5"
server_key: "281cee2e48b022f5bc2fbb4201a89be8"
server_key_iv: "868d771a2aacb6cbe78addd8a8bbd30c"
server_key_version: "encryptionkey@0"
I20260620 06:07:08.418867 16996 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781935603120653-11672-0/minicluster-data/master-0/wal/instance:
uuid: "9aba9fb589014062ad721ff283c384e0"
format_stamp: "Formatted at 2026-06-20 06:07:08 on dist-test-slave-9hm5"
server_key: "281cee2e48b022f5bc2fbb4201a89be8"
server_key_iv: "868d771a2aacb6cbe78addd8a8bbd30c"
server_key_version: "encryptionkey@0"
I20260620 06:07:08.422353 16996 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260620 06:07:08.424636 17010 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:07:08.425830 16996 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260620 06:07:08.425969 16996 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781935603120653-11672-0/minicluster-data/master-0/data,/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781935603120653-11672-0/minicluster-data/master-0/wal
uuid: "9aba9fb589014062ad721ff283c384e0"
format_stamp: "Formatted at 2026-06-20 06:07:08 on dist-test-slave-9hm5"
server_key: "281cee2e48b022f5bc2fbb4201a89be8"
server_key_iv: "868d771a2aacb6cbe78addd8a8bbd30c"
server_key_version: "encryptionkey@0"
I20260620 06:07:08.426072 16996 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781935603120653-11672-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781935603120653-11672-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781935603120653-11672-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:07:08.439894 16996 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:07:08.440551 16996 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:07:08.440742 16996 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:07:08.447167 16996 rpc_server.cc:307] RPC server started. Bound to: 127.11.102.62:44201
I20260620 06:07:08.447173 17062 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.102.62:44201 every 8 connection(s)
I20260620 06:07:08.448052 16996 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781935603120653-11672-0/minicluster-data/master-0/data/info.pb
I20260620 06:07:08.451009 17063 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 06:07:08.451829 11672 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9HMp43/build/debug/bin/kudu as pid 16996
I20260620 06:07:08.451964 11672 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781935603120653-11672-0/minicluster-data/master-0/wal/instance
I20260620 06:07:08.452244 11672 external_mini_cluster.cc:1428] Setting key 0236c404629a08df960591682b82b1c2
I20260620 06:07:08.456739 17063 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9aba9fb589014062ad721ff283c384e0: Bootstrap starting.
I20260620 06:07:08.458808 17063 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9aba9fb589014062ad721ff283c384e0: Neither blocks nor log segments found. Creating new log.
I20260620 06:07:08.459478 17063 log.cc:824] T 00000000000000000000000000000000 P 9aba9fb589014062ad721ff283c384e0: Log is configured to *not* fsync() on all Append() calls
I20260620 06:07:08.461051 17066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:07:08.453513 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51942 (local address 127.11.102.62:44201)
0620 06:07:08.454091 (+ 578us) server_negotiation.cc:206] Beginning negotiation
0620 06:07:08.454104 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 06:07:08.454136 (+ 32us) server_negotiation.cc:407] Connection header received
0620 06:07:08.454743 (+ 607us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:07:08.454758 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:07:08.455115 (+ 357us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:07:08.455442 (+ 327us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:07:08.455865 (+ 423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:07:08.456836 (+ 971us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:07:08.457721 (+ 885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:07:08.458181 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:07:08.458362 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:07:08.459199 (+ 837us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:07:08.459203 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:07:08.459215 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 06:07:08.459560 (+ 345us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:07:08.459661 (+ 101us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 06:07:08.459690 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:07:08.459790 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0620 06:07:08.459936 (+ 146us) server_negotiation.cc:299] Negotiation successful
0620 06:07:08.460184 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":269,"thread_start_us":128,"threads_started":1}
I20260620 06:07:08.461576 17063 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9aba9fb589014062ad721ff283c384e0: No bootstrap required, opened a new log
I20260620 06:07:08.463701 11672 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 06:07:08.465509 17063 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9aba9fb589014062ad721ff283c384e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9aba9fb589014062ad721ff283c384e0" member_type: VOTER last_known_addr { host: "127.11.102.62" port: 44201 } }
I20260620 06:07:08.465703 17063 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9aba9fb589014062ad721ff283c384e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 06:07:08.465793 17063 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9aba9fb589014062ad721ff283c384e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9aba9fb589014062ad721ff283c384e0, State: Initialized, Role: FOLLOWER
I20260620 06:07:08.466189 17063 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9aba9fb589014062ad721ff283c384e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9aba9fb589014062ad721ff283c384e0" member_type: VOTER last_known_addr { host: "127.11.102.62" port: 44201 } }
I20260620 06:07:08.466323 17063 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9aba9fb589014062ad721ff283c384e0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 06:07:08.466410 17063 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9aba9fb589014062ad721ff283c384e0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 06:07:08.466518 17063 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9aba9fb589014062ad721ff283c384e0 [term 0 FOLLOWER]: Advancing to term 1
I20260620 06:07:08.467489 17063 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9aba9fb589014062ad721ff283c384e0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9aba9fb589014062ad721ff283c384e0" member_type: VOTER last_known_addr { host: "127.11.102.62" port: 44201 } }
I20260620 06:07:08.467855 17063 leader_election.cc:302] T 00000000000000000000000000000000 P 9aba9fb589014062ad721ff283c384e0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9aba9fb589014062ad721ff283c384e0; no voters:
I20260620 06:07:08.468143 17063 leader_election.cc:288] T 00000000000000000000000000000000 P 9aba9fb589014062ad721ff283c384e0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 06:07:08.468281 17068 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9aba9fb589014062ad721ff283c384e0 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 06:07:08.468521 17068 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9aba9fb589014062ad721ff283c384e0 [term 1 LEADER]: Becoming Leader. State: Replica: 9aba9fb589014062ad721ff283c384e0, State: Running, Role: LEADER
I20260620 06:07:08.468813 17068 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9aba9fb589014062ad721ff283c384e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9aba9fb589014062ad721ff283c384e0" member_type: VOTER last_known_addr { host: "127.11.102.62" port: 44201 } }
I20260620 06:07:08.469307 17063 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9aba9fb589014062ad721ff283c384e0 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 06:07:08.470455 17069 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9aba9fb589014062ad721ff283c384e0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9aba9fb589014062ad721ff283c384e0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9aba9fb589014062ad721ff283c384e0" member_type: VOTER last_known_addr { host: "127.11.102.62" port: 44201 } } }
I20260620 06:07:08.470592 17069 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9aba9fb589014062ad721ff283c384e0 [sys.catalog]: This master's current role is: LEADER
I20260620 06:07:08.470851 17070 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9aba9fb589014062ad721ff283c384e0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9aba9fb589014062ad721ff283c384e0. Latest consensus state: current_term: 1 leader_uuid: "9aba9fb589014062ad721ff283c384e0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9aba9fb589014062ad721ff283c384e0" member_type: VOTER last_known_addr { host: "127.11.102.62" port: 44201 } } }
I20260620 06:07:08.470943 17070 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9aba9fb589014062ad721ff283c384e0 [sys.catalog]: This master's current role is: LEADER
I20260620 06:07:08.471078 17077 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 06:07:08.473703 17077 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 06:07:08.478302 17077 catalog_manager.cc:1347] Generated new cluster ID: 676724871ebd4929b5385c1235e4931c
I20260620 06:07:08.478361 17077 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 06:07:08.486726 17077 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 06:07:08.487967 17077 catalog_manager.cc:1504] Loading token signing keys...
I20260620 06:07:08.498329 17077 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9aba9fb589014062ad721ff283c384e0: Generated new TSK 0
I20260620 06:07:08.499006 17077 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 06:07:08.517601 17066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:07:08.512667 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34272 (local address 127.11.102.62:44201)
0620 06:07:08.512794 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0620 06:07:08.512799 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 06:07:08.512966 (+ 167us) server_negotiation.cc:407] Connection header received
0620 06:07:08.513086 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:07:08.513089 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:07:08.513143 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:07:08.513256 (+ 113us) server_negotiation.cc:226] Negotiated authn=JWT
0620 06:07:08.513446 (+ 190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:07:08.514280 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:07:08.515099 (+ 819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:07:08.515441 (+ 342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:07:08.515539 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:07:08.515852 (+ 313us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0620 06:07:08.517236 (+ 1384us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36385/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0620 06:07:08.517464 (+ 228us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:34272: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36385/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":52}
W20260620 06:07:08.517668 17066 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:34272: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36385/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260620 06:07:08.519397 17092 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 06:07:08.512562 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.102.62:44201 (local address 127.0.0.1:34272)
0620 06:07:08.512875 (+ 313us) negotiation.cc:107] Waiting for socket to connect
0620 06:07:08.512888 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 06:07:08.512987 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:07:08.513260 (+ 273us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:07:08.513262 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:07:08.513268 (+ 6us) client_negotiation.cc:189] Negotiated authn=JWT
0620 06:07:08.513355 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:07:08.513359 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:07:08.514383 (+ 1024us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:07:08.514385 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:07:08.514994 (+ 609us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:07:08.515000 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:07:08.515553 (+ 553us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:07:08.515556 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:07:08.515666 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:07:08.515688 (+ 22us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0620 06:07:08.517387 (+ 1699us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36385/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0620 06:07:08.519236 (+ 1849us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.11.102.62:44201: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36385/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":227,"mutex_wait_us":1666,"thread_start_us":108,"threads_started":1}
I20260620 06:07:08.520009 11672 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9HMp43/build/debug/bin/kudu with pid 16996
E20260620 06:07:08.528347 17093 webserver.cc:533] Webserver: sslize failed:
E20260620 06:07:08.528419 17093 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
2026-06-20T06:07:08Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (280 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-06-20T06:07:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T06:07:08Z Disabled control of system clock
I20260620 06:07:08.566305 11672 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9HMp43/build/debug/bin/kudu
/tmp/dist-test-task9HMp43/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781935603120653-11672-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781935603120653-11672-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781935603120653-11672-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781935603120653-11672-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.102.62:46311
--webserver_interface=127.11.102.62
--webserver_port=0
--builtin_ntp_servers=127.11.102.20:35715
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.102.62:46311
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781935603120653-11672-0/testchainca.cert with env {}
W20260620 06:07:08.671627 17097 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:07:08.671905 17097 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:07:08.671983 17097 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:07:08.675947 17097 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 06:07:08.676018 17097 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:07:08.676046 17097 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:07:08.676096 17097 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 06:07:08.676141 17097 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260620 06:07:08.676180 17097 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260620 06:07:08.676213 17097 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260620 06:07:08.681290 17097 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.102.20:35715
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781935603120653-11672-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781935603120653-11672-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.102.62:46311
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.102.62:46311
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781935603120653-11672-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.102.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781935603120653-11672-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17097
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781935603120653-11672-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 05:28:48 UTC on ccycloud.quasar-fixhms.root.comops.site
build id 4457020
I20260620 06:07:08.682247 17097 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:07:08.683177 17097 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 06:07:08.691414 17105 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:07:08.691448 17097 server_base.cc:1034] running on GCE node
W20260620 06:07:08.691390 17103 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:07:08.691404 17102 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:07:08.692251 17097 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:07:08.693339 17097 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:07:08.694530 17097 hybrid_clock.cc:648] HybridClock initialized: now 1781935628694513 us; error 31 us; skew 500 ppm
I20260620 06:07:08.696388 17097 webserver.cc:460] Webserver started at http://127.11.102.62:36237/ using document root <none> and password file <none>
I20260620 06:07:08.696970 17097 fs_manager.cc:359] Metadata directory not provided
I20260620 06:07:08.697063 17097 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:07:08.697288 17097 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 06:07:08.699273 17097 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781935603120653-11672-0/minicluster-data/master-0/data/instance:
uuid: "6fd7f96a421a4ea7bb1fca887ddda6ed"
format_stamp: "Formatted at 2026-06-20 06:07:08 on dist-test-slave-9hm5"
server_key: "c045dddbf6c3460d7e32d81bb172db13"
server_key_iv: "10b55744496af2c5aba63a97d6dbdf63"
server_key_version: "encryptionkey@0"
I20260620 06:07:08.699869 17097 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781935603120653-11672-0/minicluster-data/master-0/wal/instance:
uuid: "6fd7f96a421a4ea7bb1fca887ddda6ed"
format_stamp: "Formatted at 2026-06-20 06:07:08 on dist-test-slave-9hm5"
server_key: "c045dddbf6c3460d7e32d81bb172db13"
server_key_iv: "10b55744496af2c5aba63a97d6dbdf63"
server_key_version: "encryptionkey@0"
I20260620 06:07:08.703338 17097 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260620 06:07:08.705629 17111 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:07:08.706768 17097 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260620 06:07:08.706920 17097 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781935603120653-11672-0/minicluster-data/master-0/data,/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781935603120653-11672-0/minicluster-data/master-0/wal
uuid: "6fd7f96a421a4ea7bb1fca887ddda6ed"
format_stamp: "Formatted at 2026-06-20 06:07:08 on dist-test-slave-9hm5"
server_key: "c045dddbf6c3460d7e32d81bb172db13"
server_key_iv: "10b55744496af2c5aba63a97d6dbdf63"
server_key_version: "encryptionkey@0"
I20260620 06:07:08.707026 17097 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781935603120653-11672-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781935603120653-11672-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781935603120653-11672-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:07:08.725612 17097 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:07:08.726285 17097 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:07:08.726480 17097 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:07:08.733274 17097 rpc_server.cc:307] RPC server started. Bound to: 127.11.102.62:46311
I20260620 06:07:08.733294 17163 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.102.62:46311 every 8 connection(s)
I20260620 06:07:08.734179 17097 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781935603120653-11672-0/minicluster-data/master-0/data/info.pb
I20260620 06:07:08.736835 17164 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 06:07:08.741536 11672 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9HMp43/build/debug/bin/kudu as pid 17097
I20260620 06:07:08.741676 11672 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781935603120653-11672-0/minicluster-data/master-0/wal/instance
I20260620 06:07:08.741948 11672 external_mini_cluster.cc:1428] Setting key ea6ff7f1dce96c275418f2319b58f139
I20260620 06:07:08.742095 17164 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6fd7f96a421a4ea7bb1fca887ddda6ed: Bootstrap starting.
I20260620 06:07:08.744673 17164 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6fd7f96a421a4ea7bb1fca887ddda6ed: Neither blocks nor log segments found. Creating new log.
I20260620 06:07:08.745481 17164 log.cc:824] T 00000000000000000000000000000000 P 6fd7f96a421a4ea7bb1fca887ddda6ed: Log is configured to *not* fsync() on all Append() calls
I20260620 06:07:08.747447 17164 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6fd7f96a421a4ea7bb1fca887ddda6ed: No bootstrap required, opened a new log
I20260620 06:07:08.751485 17164 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6fd7f96a421a4ea7bb1fca887ddda6ed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6fd7f96a421a4ea7bb1fca887ddda6ed" member_type: VOTER last_known_addr { host: "127.11.102.62" port: 46311 } }
I20260620 06:07:08.751737 17164 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6fd7f96a421a4ea7bb1fca887ddda6ed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 06:07:08.751796 17164 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6fd7f96a421a4ea7bb1fca887ddda6ed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6fd7f96a421a4ea7bb1fca887ddda6ed, State: Initialized, Role: FOLLOWER
I20260620 06:07:08.751995 17167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:07:08.743216 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45700 (local address 127.11.102.62:46311)
0620 06:07:08.743909 (+ 693us) server_negotiation.cc:206] Beginning negotiation
0620 06:07:08.743927 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0620 06:07:08.743968 (+ 41us) server_negotiation.cc:407] Connection header received
0620 06:07:08.744793 (+ 825us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:07:08.744814 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:07:08.745293 (+ 479us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:07:08.745648 (+ 355us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:07:08.746026 (+ 378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:07:08.747384 (+ 1358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:07:08.748351 (+ 967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:07:08.749037 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:07:08.749200 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:07:08.749937 (+ 737us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:07:08.749943 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:07:08.749958 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 06:07:08.750483 (+ 525us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:07:08.750559 (+ 76us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 06:07:08.750603 (+ 44us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:07:08.750709 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0620 06:07:08.750824 (+ 115us) server_negotiation.cc:299] Negotiation successful
0620 06:07:08.751095 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":272,"thread_start_us":113,"threads_started":1}
I20260620 06:07:08.752240 17164 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6fd7f96a421a4ea7bb1fca887ddda6ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6fd7f96a421a4ea7bb1fca887ddda6ed" member_type: VOTER last_known_addr { host: "127.11.102.62" port: 46311 } }
I20260620 06:07:08.752385 17164 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6fd7f96a421a4ea7bb1fca887ddda6ed [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 06:07:08.752446 17164 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6fd7f96a421a4ea7bb1fca887ddda6ed [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 06:07:08.752525 17164 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6fd7f96a421a4ea7bb1fca887ddda6ed [term 0 FOLLOWER]: Advancing to term 1
I20260620 06:07:08.753404 17164 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6fd7f96a421a4ea7bb1fca887ddda6ed [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6fd7f96a421a4ea7bb1fca887ddda6ed" member_type: VOTER last_known_addr { host: "127.11.102.62" port: 46311 } }
I20260620 06:07:08.753737 17164 leader_election.cc:302] T 00000000000000000000000000000000 P 6fd7f96a421a4ea7bb1fca887ddda6ed [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6fd7f96a421a4ea7bb1fca887ddda6ed; no voters:
I20260620 06:07:08.754029 17164 leader_election.cc:288] T 00000000000000000000000000000000 P 6fd7f96a421a4ea7bb1fca887ddda6ed [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 06:07:08.754153 17169 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6fd7f96a421a4ea7bb1fca887ddda6ed [term 1 FOLLOWER]: Leader election won for term 1
I20260620 06:07:08.754356 17169 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6fd7f96a421a4ea7bb1fca887ddda6ed [term 1 LEADER]: Becoming Leader. State: Replica: 6fd7f96a421a4ea7bb1fca887ddda6ed, State: Running, Role: LEADER
I20260620 06:07:08.754689 11672 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 06:07:08.754647 17169 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6fd7f96a421a4ea7bb1fca887ddda6ed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6fd7f96a421a4ea7bb1fca887ddda6ed" member_type: VOTER last_known_addr { host: "127.11.102.62" port: 46311 } }
I20260620 06:07:08.755223 17164 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6fd7f96a421a4ea7bb1fca887ddda6ed [sys.catalog]: configured and running, proceeding with master startup.
I20260620 06:07:08.756124 17170 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6fd7f96a421a4ea7bb1fca887ddda6ed [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6fd7f96a421a4ea7bb1fca887ddda6ed. Latest consensus state: current_term: 1 leader_uuid: "6fd7f96a421a4ea7bb1fca887ddda6ed" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6fd7f96a421a4ea7bb1fca887ddda6ed" member_type: VOTER last_known_addr { host: "127.11.102.62" port: 46311 } } }
I20260620 06:07:08.756254 17170 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6fd7f96a421a4ea7bb1fca887ddda6ed [sys.catalog]: This master's current role is: LEADER
I20260620 06:07:08.756518 17171 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6fd7f96a421a4ea7bb1fca887ddda6ed [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6fd7f96a421a4ea7bb1fca887ddda6ed" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6fd7f96a421a4ea7bb1fca887ddda6ed" member_type: VOTER last_known_addr { host: "127.11.102.62" port: 46311 } } }
I20260620 06:07:08.756615 17171 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6fd7f96a421a4ea7bb1fca887ddda6ed [sys.catalog]: This master's current role is: LEADER
W20260620 06:07:08.759773 17184 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 6fd7f96a421a4ea7bb1fca887ddda6ed: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 06:07:08.759848 17184 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 06:07:08.759923 17178 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 06:07:08.760565 17178 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 06:07:08.765471 17178 catalog_manager.cc:1347] Generated new cluster ID: b3cf678a7408471da8ede1959f289f49
I20260620 06:07:08.765547 17178 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 06:07:08.790124 17178 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 06:07:08.791142 17178 catalog_manager.cc:1504] Loading token signing keys...
I20260620 06:07:08.802801 17178 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6fd7f96a421a4ea7bb1fca887ddda6ed: Generated new TSK 0
I20260620 06:07:08.803567 17178 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 06:07:08.879519 17167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:07:08.814737 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45708 (local address 127.11.102.62:46311)
0620 06:07:08.814856 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0620 06:07:08.814861 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 06:07:08.815026 (+ 165us) server_negotiation.cc:407] Connection header received
0620 06:07:08.815231 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:07:08.815233 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:07:08.815279 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:07:08.815387 (+ 108us) server_negotiation.cc:226] Negotiated authn=JWT
0620 06:07:08.815636 (+ 249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:07:08.816163 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:07:08.817024 (+ 861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:07:08.817259 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:07:08.817371 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:07:08.817682 (+ 311us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0620 06:07:08.879026 (+ 61344us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0620 06:07:08.879367 (+ 341us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:45708: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":42}
W20260620 06:07:08.879638 17167 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:45708: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260620 06:07:08.879699 17193 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 06:07:08.814644 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.102.62:46311 (local address 127.0.0.1:45708)
0620 06:07:08.814933 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0620 06:07:08.814953 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 06:07:08.815073 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:07:08.815384 (+ 311us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:07:08.815387 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:07:08.815394 (+ 7us) client_negotiation.cc:189] Negotiated authn=JWT
0620 06:07:08.815500 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:07:08.815506 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:07:08.816276 (+ 770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:07:08.816281 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:07:08.816905 (+ 624us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:07:08.816912 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:07:08.817387 (+ 475us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:07:08.817390 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:07:08.817490 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:07:08.817510 (+ 20us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0620 06:07:08.879280 (+ 61770us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0620 06:07:08.879468 (+ 188us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.11.102.62:46311: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":214,"thread_start_us":115,"threads_started":1}
I20260620 06:07:08.880121 11672 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9HMp43/build/debug/bin/kudu with pid 17097
2026-06-20T06:07:08Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (341 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (119 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (118 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781935603120653-11672-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 06:07:09 dist-test-slave-9hm5 krb5kdc[17205](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 06:07:09 dist-test-slave-9hm5 krb5kdc[17205](info): set up 2 sockets
Jun 20 06:07:09 dist-test-slave-9hm5 krb5kdc[17205](info): commencing operation
krb5kdc: starting...
W20260620 06:07:10.168092 11672 mini_kdc.cc:121] Time spent starting KDC: real 1.030s user 0.005s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781935603120653-11672-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781935603120653-11672-0/krb5kdc/test-user.keytab.
Jun 20 06:07:10 dist-test-slave-9hm5 krb5kdc[17205](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935630, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T06:07:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T06:07:10Z Disabled control of system clock
WARNING: no policy specified for kudu/127.11.102.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.102.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.102.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781935603120653-11672-0/krb5kdc/kudu_127.11.102.62.keytab.
Entry for principal kudu/127.11.102.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781935603120653-11672-0/krb5kdc/kudu_127.11.102.62.keytab.
I20260620 06:07:10.285809 11672 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9HMp43/build/debug/bin/kudu
/tmp/dist-test-task9HMp43/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781935603120653-11672-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781935603120653-11672-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781935603120653-11672-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781935603120653-11672-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.102.62:44137
--webserver_interface=127.11.102.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781935603120653-11672-0/krb5kdc/kudu_127.11.102.62.keytab
--principal=kudu/127.11.102.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.102.20:33757
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.102.62:44137
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781935603120653-11672-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781935603120653-11672-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781935603120653-11672-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 06:07:10.392818 17218 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:07:10.393118 17218 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:07:10.393191 17218 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:07:10.397159 17218 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 06:07:10.397251 17218 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:07:10.397290 17218 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:07:10.397330 17218 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 06:07:10.397361 17218 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 06:07:10.402449 17218 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.102.20:33757
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781935603120653-11672-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781935603120653-11672-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.102.62:44137
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781935603120653-11672-0/krb5kdc/kudu_127.11.102.62.keytab
--principal=kudu/127.11.102.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.102.62:44137
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781935603120653-11672-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.102.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17218
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781935603120653-11672-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 05:28:48 UTC on ccycloud.quasar-fixhms.root.comops.site
build id 4457020
I20260620 06:07:10.403482 17218 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:07:10.404583 17218 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 06:07:10.412668 17224 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:07:10.412683 17226 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:07:10.412782 17218 server_base.cc:1034] running on GCE node
W20260620 06:07:10.412765 17223 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:07:10.413430 17218 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:07:10.414472 17218 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:07:10.415714 17218 hybrid_clock.cc:648] HybridClock initialized: now 1781935630415691 us; error 35 us; skew 500 ppm
Jun 20 06:07:10 dist-test-slave-9hm5 krb5kdc[17205](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935630, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.102.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 06:07:10.418386 17218 init.cc:377] Logged in from keytab as kudu/127.11.102.62@KRBTEST.COM (short username kudu)
I20260620 06:07:10.419194 17218 webserver.cc:460] Webserver started at http://127.11.102.62:33229/ using document root <none> and password file <none>
I20260620 06:07:10.419790 17218 fs_manager.cc:359] Metadata directory not provided
I20260620 06:07:10.419879 17218 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:07:10.420101 17218 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 06:07:10.422108 17218 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781935603120653-11672-0/minicluster-data/master-0/data/instance:
uuid: "2ff3a45dbcc84e2f8da196f2fcb2a50c"
format_stamp: "Formatted at 2026-06-20 06:07:10 on dist-test-slave-9hm5"
server_key: "7b52f615298c63179724c8804accb5ea"
server_key_iv: "58c27b64ddb9062b5ae8f916defc52bb"
server_key_version: "encryptionkey@0"
I20260620 06:07:10.422657 17218 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781935603120653-11672-0/minicluster-data/master-0/wal/instance:
uuid: "2ff3a45dbcc84e2f8da196f2fcb2a50c"
format_stamp: "Formatted at 2026-06-20 06:07:10 on dist-test-slave-9hm5"
server_key: "7b52f615298c63179724c8804accb5ea"
server_key_iv: "58c27b64ddb9062b5ae8f916defc52bb"
server_key_version: "encryptionkey@0"
I20260620 06:07:10.426198 17218 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.003s
I20260620 06:07:10.428514 17233 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:07:10.429678 17218 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 06:07:10.429813 17218 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781935603120653-11672-0/minicluster-data/master-0/data,/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781935603120653-11672-0/minicluster-data/master-0/wal
uuid: "2ff3a45dbcc84e2f8da196f2fcb2a50c"
format_stamp: "Formatted at 2026-06-20 06:07:10 on dist-test-slave-9hm5"
server_key: "7b52f615298c63179724c8804accb5ea"
server_key_iv: "58c27b64ddb9062b5ae8f916defc52bb"
server_key_version: "encryptionkey@0"
I20260620 06:07:10.429919 17218 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781935603120653-11672-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781935603120653-11672-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781935603120653-11672-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:07:10.442236 17218 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:07:10.445021 17218 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:07:10.445212 17218 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:07:10.451525 17218 rpc_server.cc:307] RPC server started. Bound to: 127.11.102.62:44137
I20260620 06:07:10.451643 17285 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.102.62:44137 every 8 connection(s)
I20260620 06:07:10.452476 17218 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781935603120653-11672-0/minicluster-data/master-0/data/info.pb
I20260620 06:07:10.455071 17286 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 06:07:10.460304 17286 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2ff3a45dbcc84e2f8da196f2fcb2a50c: Bootstrap starting.
I20260620 06:07:10.461311 11672 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9HMp43/build/debug/bin/kudu as pid 17218
I20260620 06:07:10.461417 11672 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781935603120653-11672-0/minicluster-data/master-0/wal/instance
I20260620 06:07:10.461709 11672 external_mini_cluster.cc:1428] Setting key 5178dc3f03a6493dbd0ee2aa60e69fc0
I20260620 06:07:10.462960 17286 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2ff3a45dbcc84e2f8da196f2fcb2a50c: Neither blocks nor log segments found. Creating new log.
I20260620 06:07:10.463692 17286 log.cc:824] T 00000000000000000000000000000000 P 2ff3a45dbcc84e2f8da196f2fcb2a50c: Log is configured to *not* fsync() on all Append() calls
I20260620 06:07:10.465843 17286 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2ff3a45dbcc84e2f8da196f2fcb2a50c: No bootstrap required, opened a new log
I20260620 06:07:10.470247 17286 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2ff3a45dbcc84e2f8da196f2fcb2a50c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ff3a45dbcc84e2f8da196f2fcb2a50c" member_type: VOTER last_known_addr { host: "127.11.102.62" port: 44137 } }
I20260620 06:07:10.470479 17286 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2ff3a45dbcc84e2f8da196f2fcb2a50c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 06:07:10.470567 17286 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2ff3a45dbcc84e2f8da196f2fcb2a50c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2ff3a45dbcc84e2f8da196f2fcb2a50c, State: Initialized, Role: FOLLOWER
Jun 20 06:07:10 dist-test-slave-9hm5 krb5kdc[17205](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935630, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.102.62@KRBTEST.COM
I20260620 06:07:10.471079 17286 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2ff3a45dbcc84e2f8da196f2fcb2a50c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ff3a45dbcc84e2f8da196f2fcb2a50c" member_type: VOTER last_known_addr { host: "127.11.102.62" port: 44137 } }
I20260620 06:07:10.471254 17286 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2ff3a45dbcc84e2f8da196f2fcb2a50c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 06:07:10.471325 17286 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2ff3a45dbcc84e2f8da196f2fcb2a50c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 06:07:10.471422 17286 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2ff3a45dbcc84e2f8da196f2fcb2a50c [term 0 FOLLOWER]: Advancing to term 1
I20260620 06:07:10.472561 17286 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2ff3a45dbcc84e2f8da196f2fcb2a50c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ff3a45dbcc84e2f8da196f2fcb2a50c" member_type: VOTER last_known_addr { host: "127.11.102.62" port: 44137 } }
I20260620 06:07:10.472903 17286 leader_election.cc:302] T 00000000000000000000000000000000 P 2ff3a45dbcc84e2f8da196f2fcb2a50c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2ff3a45dbcc84e2f8da196f2fcb2a50c; no voters:
I20260620 06:07:10.473230 17286 leader_election.cc:288] T 00000000000000000000000000000000 P 2ff3a45dbcc84e2f8da196f2fcb2a50c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 06:07:10.473363 17291 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2ff3a45dbcc84e2f8da196f2fcb2a50c [term 1 FOLLOWER]: Leader election won for term 1
I20260620 06:07:10.473529 17291 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2ff3a45dbcc84e2f8da196f2fcb2a50c [term 1 LEADER]: Becoming Leader. State: Replica: 2ff3a45dbcc84e2f8da196f2fcb2a50c, State: Running, Role: LEADER
I20260620 06:07:10.473882 17291 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2ff3a45dbcc84e2f8da196f2fcb2a50c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ff3a45dbcc84e2f8da196f2fcb2a50c" member_type: VOTER last_known_addr { host: "127.11.102.62" port: 44137 } }
I20260620 06:07:10.474738 17286 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2ff3a45dbcc84e2f8da196f2fcb2a50c [sys.catalog]: configured and running, proceeding with master startup.
I20260620 06:07:10.475783 17292 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2ff3a45dbcc84e2f8da196f2fcb2a50c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2ff3a45dbcc84e2f8da196f2fcb2a50c. Latest consensus state: current_term: 1 leader_uuid: "2ff3a45dbcc84e2f8da196f2fcb2a50c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ff3a45dbcc84e2f8da196f2fcb2a50c" member_type: VOTER last_known_addr { host: "127.11.102.62" port: 44137 } } }
I20260620 06:07:10.475911 17292 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2ff3a45dbcc84e2f8da196f2fcb2a50c [sys.catalog]: This master's current role is: LEADER
I20260620 06:07:10.475783 17293 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2ff3a45dbcc84e2f8da196f2fcb2a50c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2ff3a45dbcc84e2f8da196f2fcb2a50c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ff3a45dbcc84e2f8da196f2fcb2a50c" member_type: VOTER last_known_addr { host: "127.11.102.62" port: 44137 } } }
I20260620 06:07:10.476240 17293 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2ff3a45dbcc84e2f8da196f2fcb2a50c [sys.catalog]: This master's current role is: LEADER
I20260620 06:07:10.476356 17300 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 06:07:10.477175 17289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:07:10.462834 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59748 (local address 127.11.102.62:44137)
0620 06:07:10.463246 (+ 412us) server_negotiation.cc:206] Beginning negotiation
0620 06:07:10.463261 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 06:07:10.463300 (+ 39us) server_negotiation.cc:407] Connection header received
0620 06:07:10.464005 (+ 705us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:07:10.464023 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:07:10.464378 (+ 355us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:07:10.464645 (+ 267us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:07:10.465523 (+ 878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:07:10.466537 (+ 1014us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:07:10.467764 (+ 1227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:07:10.468221 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:07:10.468421 (+ 200us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:07:10.471233 (+ 2812us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:07:10.471257 (+ 24us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:07:10.471274 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:07:10.471344 (+ 70us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:07:10.473216 (+ 1872us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:07:10.473756 (+ 540us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:07:10.473763 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:07:10.473770 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:07:10.473916 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:07:10.474229 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:07:10.474235 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:07:10.474238 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:07:10.474727 (+ 489us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:07:10.474913 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0620 06:07:10.475379 (+ 466us) server_negotiation.cc:299] Negotiation successful
0620 06:07:10.475727 (+ 348us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":267,"thread_start_us":130,"threads_started":1}
I20260620 06:07:10.479841 17300 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 06:07:10.484930 17300 catalog_manager.cc:1347] Generated new cluster ID: 38758fc66225430883295f68729ca913
I20260620 06:07:10.484992 17300 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 06:07:10.496328 17300 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 06:07:10.497535 17300 catalog_manager.cc:1504] Loading token signing keys...
I20260620 06:07:10.509344 17300 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2ff3a45dbcc84e2f8da196f2fcb2a50c: Generated new TSK 0
I20260620 06:07:10.510051 17300 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.11.102.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.102.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.102.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781935603120653-11672-0/krb5kdc/kudu_127.11.102.1.keytab.
Entry for principal kudu/127.11.102.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781935603120653-11672-0/krb5kdc/kudu_127.11.102.1.keytab.
I20260620 06:07:10.547192 11672 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9HMp43/build/debug/bin/kudu
/tmp/dist-test-task9HMp43/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781935603120653-11672-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781935603120653-11672-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781935603120653-11672-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781935603120653-11672-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.102.1:0
--local_ip_for_outbound_sockets=127.11.102.1
--webserver_interface=127.11.102.1
--webserver_port=0
--tserver_master_addrs=127.11.102.62:44137
--keytab_file=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781935603120653-11672-0/krb5kdc/kudu_127.11.102.1.keytab
--principal=kudu/127.11.102.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.102.20:33757
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781935603120653-11672-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781935603120653-11672-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781935603120653-11672-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 06:07:10.661242 17312 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:07:10.661474 17312 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:07:10.661513 17312 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:07:10.665299 17312 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:07:10.665342 17312 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:07:10.665395 17312 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.102.1
I20260620 06:07:10.670456 17312 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.102.20:33757
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781935603120653-11672-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781935603120653-11672-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781935603120653-11672-0/krb5kdc/kudu_127.11.102.1.keytab
--principal=kudu/127.11.102.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.102.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781935603120653-11672-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.11.102.1
--webserver_port=0
--tserver_master_addrs=127.11.102.62:44137
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17312
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.102.1
--log_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781935603120653-11672-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 05:28:48 UTC on ccycloud.quasar-fixhms.root.comops.site
build id 4457020
I20260620 06:07:10.671332 17312 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:07:10.672250 17312 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 06:07:10.681687 17317 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:07:10.681675 17318 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:07:10.681948 17312 server_base.cc:1034] running on GCE node
W20260620 06:07:10.681665 17320 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:07:10.682482 17312 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:07:10.683084 17312 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:07:10.684244 17312 hybrid_clock.cc:648] HybridClock initialized: now 1781935630684226 us; error 27 us; skew 500 ppm
Jun 20 06:07:10 dist-test-slave-9hm5 krb5kdc[17205](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935630, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.102.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 06:07:10.686723 17312 init.cc:377] Logged in from keytab as kudu/127.11.102.1@KRBTEST.COM (short username kudu)
I20260620 06:07:10.687506 17312 webserver.cc:460] Webserver started at http://127.11.102.1:39185/ using document root <none> and password file <none>
I20260620 06:07:10.688064 17312 fs_manager.cc:359] Metadata directory not provided
I20260620 06:07:10.688155 17312 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:07:10.688388 17312 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 06:07:10.690276 17312 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781935603120653-11672-0/minicluster-data/ts-0/data/instance:
uuid: "bf15e3915a2f47888ddd64400f286cc0"
format_stamp: "Formatted at 2026-06-20 06:07:10 on dist-test-slave-9hm5"
server_key: "87fb0031053237d711bf7caef16232a9"
server_key_iv: "33a4841203e78009998241a481deaebd"
server_key_version: "encryptionkey@0"
I20260620 06:07:10.690799 17312 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781935603120653-11672-0/minicluster-data/ts-0/wal/instance:
uuid: "bf15e3915a2f47888ddd64400f286cc0"
format_stamp: "Formatted at 2026-06-20 06:07:10 on dist-test-slave-9hm5"
server_key: "87fb0031053237d711bf7caef16232a9"
server_key_iv: "33a4841203e78009998241a481deaebd"
server_key_version: "encryptionkey@0"
I20260620 06:07:10.694406 17312 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260620 06:07:10.696794 17327 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:07:10.697975 17312 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260620 06:07:10.698087 17312 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781935603120653-11672-0/minicluster-data/ts-0/data,/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781935603120653-11672-0/minicluster-data/ts-0/wal
uuid: "bf15e3915a2f47888ddd64400f286cc0"
format_stamp: "Formatted at 2026-06-20 06:07:10 on dist-test-slave-9hm5"
server_key: "87fb0031053237d711bf7caef16232a9"
server_key_iv: "33a4841203e78009998241a481deaebd"
server_key_version: "encryptionkey@0"
I20260620 06:07:10.698181 17312 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781935603120653-11672-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781935603120653-11672-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781935603120653-11672-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:07:10.723583 17312 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:07:10.726284 17312 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:07:10.726469 17312 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:07:10.727015 17312 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 06:07:10.727777 17312 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 06:07:10.727818 17312 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 06:07:10.727880 17312 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 06:07:10.727908 17312 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 06:07:10.736986 17312 rpc_server.cc:307] RPC server started. Bound to: 127.11.102.1:40349
I20260620 06:07:10.737010 17440 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.102.1:40349 every 8 connection(s)
I20260620 06:07:10.737779 17312 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781935603120653-11672-0/minicluster-data/ts-0/data/info.pb
I20260620 06:07:10.743402 11672 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9HMp43/build/debug/bin/kudu as pid 17312
I20260620 06:07:10.743505 11672 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781935603120653-11672-0/minicluster-data/ts-0/wal/instance
I20260620 06:07:10.743819 11672 external_mini_cluster.cc:1428] Setting key add12a1b2f181dfd3b955684db481883
Jun 20 06:07:10 dist-test-slave-9hm5 krb5kdc[17205](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935630, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.102.1@KRBTEST.COM for kudu/127.11.102.62@KRBTEST.COM
I20260620 06:07:10.751151 17289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:07:10.739517 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.102.1:57967 (local address 127.11.102.62:44137)
0620 06:07:10.739655 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0620 06:07:10.739660 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 06:07:10.740480 (+ 820us) server_negotiation.cc:407] Connection header received
0620 06:07:10.741708 (+ 1228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:07:10.741711 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:07:10.741759 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:07:10.741835 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:07:10.742904 (+ 1069us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:07:10.743474 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:07:10.744905 (+ 1431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:07:10.745260 (+ 355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:07:10.745366 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:07:10.747693 (+ 2327us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:07:10.747702 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:07:10.747704 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:07:10.747726 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:07:10.749156 (+ 1430us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:07:10.749683 (+ 527us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:07:10.749687 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:07:10.749688 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:07:10.749765 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:07:10.750126 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:07:10.750129 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:07:10.750131 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:07:10.750298 (+ 167us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:07:10.750402 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0620 06:07:10.750863 (+ 461us) server_negotiation.cc:299] Negotiation successful
0620 06:07:10.750975 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260620 06:07:10.751938 17443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:07:10.739891 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.102.62:44137 (local address 127.11.102.1:57967)
0620 06:07:10.740328 (+ 437us) negotiation.cc:107] Waiting for socket to connect
0620 06:07:10.740367 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0620 06:07:10.741512 (+ 1145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:07:10.741986 (+ 474us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:07:10.741992 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:07:10.742320 (+ 328us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:07:10.742750 (+ 430us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:07:10.742763 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:07:10.743601 (+ 838us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:07:10.743633 (+ 32us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:07:10.744795 (+ 1162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:07:10.744805 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:07:10.745404 (+ 599us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:07:10.745407 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:07:10.745540 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:07:10.746107 (+ 567us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:07:10.746122 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:07:10.747503 (+ 1381us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:07:10.749284 (+ 1781us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:07:10.749289 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:07:10.749302 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:07:10.749571 (+ 269us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:07:10.749880 (+ 309us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:07:10.749883 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:07:10.749885 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:07:10.750017 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:07:10.750400 (+ 383us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:07:10.750407 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:07:10.750636 (+ 229us) client_negotiation.cc:769] Sending connection context
0620 06:07:10.750807 (+ 171us) client_negotiation.cc:240] Negotiation successful
0620 06:07:10.751093 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":263,"thread_start_us":107,"threads_started":1}
WARNING: no policy specified for kudu/127.11.102.2@KRBTEST.COM; defaulting to no policy
I20260620 06:07:10.753099 17441 heartbeater.cc:347] Connected to a master server at 127.11.102.62:44137
I20260620 06:07:10.753332 17441 heartbeater.cc:464] Registering TS with master...
I20260620 06:07:10.753784 17441 heartbeater.cc:511] Master 127.11.102.62:44137 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.102.2@KRBTEST.COM" created.
I20260620 06:07:10.755268 17250 ts_manager.cc:194] Registered new tserver with Master: bf15e3915a2f47888ddd64400f286cc0 (127.11.102.1:40349)
I20260620 06:07:10.756901 17250 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.102.1@KRBTEST.COM'} at 127.11.102.1:57967
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.102.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781935603120653-11672-0/krb5kdc/kudu_127.11.102.2.keytab.
Entry for principal kudu/127.11.102.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781935603120653-11672-0/krb5kdc/kudu_127.11.102.2.keytab.
I20260620 06:07:10.770104 11672 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9HMp43/build/debug/bin/kudu
/tmp/dist-test-task9HMp43/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781935603120653-11672-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781935603120653-11672-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781935603120653-11672-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781935603120653-11672-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.102.2:0
--local_ip_for_outbound_sockets=127.11.102.2
--webserver_interface=127.11.102.2
--webserver_port=0
--tserver_master_addrs=127.11.102.62:44137
--keytab_file=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781935603120653-11672-0/krb5kdc/kudu_127.11.102.2.keytab
--principal=kudu/127.11.102.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.102.20:33757
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781935603120653-11672-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781935603120653-11672-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781935603120653-11672-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 06:07:10.875007 17446 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:07:10.875272 17446 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:07:10.875314 17446 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:07:10.879518 17446 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:07:10.879580 17446 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:07:10.879709 17446 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.102.2
I20260620 06:07:10.884918 17446 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.102.20:33757
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781935603120653-11672-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781935603120653-11672-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781935603120653-11672-0/krb5kdc/kudu_127.11.102.2.keytab
--principal=kudu/127.11.102.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.102.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781935603120653-11672-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.11.102.2
--webserver_port=0
--tserver_master_addrs=127.11.102.62:44137
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17446
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.102.2
--log_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781935603120653-11672-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 05:28:48 UTC on ccycloud.quasar-fixhms.root.comops.site
build id 4457020
I20260620 06:07:10.885921 17446 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:07:10.886888 17446 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 06:07:10.896242 17451 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:07:10.896451 17446 server_base.cc:1034] running on GCE node
W20260620 06:07:10.896579 17452 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:07:10.896358 17454 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:07:10.897133 17446 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:07:10.897934 17446 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:07:10.899113 17446 hybrid_clock.cc:648] HybridClock initialized: now 1781935630899067 us; error 73 us; skew 500 ppm
Jun 20 06:07:10 dist-test-slave-9hm5 krb5kdc[17205](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935630, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.102.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 06:07:10.901863 17446 init.cc:377] Logged in from keytab as kudu/127.11.102.2@KRBTEST.COM (short username kudu)
I20260620 06:07:10.902665 17446 webserver.cc:460] Webserver started at http://127.11.102.2:37781/ using document root <none> and password file <none>
I20260620 06:07:10.903203 17446 fs_manager.cc:359] Metadata directory not provided
I20260620 06:07:10.903263 17446 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:07:10.903513 17446 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 06:07:10.905535 17446 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781935603120653-11672-0/minicluster-data/ts-1/data/instance:
uuid: "da49668cf7894333bc41bd9502fc0485"
format_stamp: "Formatted at 2026-06-20 06:07:10 on dist-test-slave-9hm5"
server_key: "29be2c870be1b7a9719862874d637730"
server_key_iv: "f7c79f9b84c0528fb5e8c3761055fe14"
server_key_version: "encryptionkey@0"
I20260620 06:07:10.906083 17446 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781935603120653-11672-0/minicluster-data/ts-1/wal/instance:
uuid: "da49668cf7894333bc41bd9502fc0485"
format_stamp: "Formatted at 2026-06-20 06:07:10 on dist-test-slave-9hm5"
server_key: "29be2c870be1b7a9719862874d637730"
server_key_iv: "f7c79f9b84c0528fb5e8c3761055fe14"
server_key_version: "encryptionkey@0"
I20260620 06:07:10.909886 17446 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260620 06:07:10.912396 17461 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:07:10.913610 17446 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 06:07:10.913745 17446 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781935603120653-11672-0/minicluster-data/ts-1/data,/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781935603120653-11672-0/minicluster-data/ts-1/wal
uuid: "da49668cf7894333bc41bd9502fc0485"
format_stamp: "Formatted at 2026-06-20 06:07:10 on dist-test-slave-9hm5"
server_key: "29be2c870be1b7a9719862874d637730"
server_key_iv: "f7c79f9b84c0528fb5e8c3761055fe14"
server_key_version: "encryptionkey@0"
I20260620 06:07:10.913846 17446 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781935603120653-11672-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781935603120653-11672-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781935603120653-11672-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:07:10.938776 17446 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:07:10.941708 17446 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:07:10.941907 17446 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:07:10.942483 17446 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 06:07:10.943291 17446 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 06:07:10.943337 17446 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 06:07:10.943403 17446 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 06:07:10.943446 17446 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 06:07:10.951962 17446 rpc_server.cc:307] RPC server started. Bound to: 127.11.102.2:37599
I20260620 06:07:10.951994 17574 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.102.2:37599 every 8 connection(s)
I20260620 06:07:10.952832 17446 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781935603120653-11672-0/minicluster-data/ts-1/data/info.pb
I20260620 06:07:10.955699 11672 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9HMp43/build/debug/bin/kudu as pid 17446
I20260620 06:07:10.955821 11672 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781935603120653-11672-0/minicluster-data/ts-1/wal/instance
I20260620 06:07:10.956162 11672 external_mini_cluster.cc:1428] Setting key 039406ad21cb9d835bb248ad67495d1a
Jun 20 06:07:10 dist-test-slave-9hm5 krb5kdc[17205](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935630, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.102.2@KRBTEST.COM for kudu/127.11.102.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.11.102.3@KRBTEST.COM; defaulting to no policy
I20260620 06:07:10.967341 17289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:07:10.954470 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.102.2:37271 (local address 127.11.102.62:44137)
0620 06:07:10.954628 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0620 06:07:10.954633 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 06:07:10.955517 (+ 884us) server_negotiation.cc:407] Connection header received
0620 06:07:10.957200 (+ 1683us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:07:10.957203 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:07:10.957258 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:07:10.957330 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:07:10.958807 (+ 1477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:07:10.959434 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:07:10.960647 (+ 1213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:07:10.960899 (+ 252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:07:10.961032 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:07:10.963756 (+ 2724us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:07:10.963759 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:07:10.963761 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:07:10.963785 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:07:10.965252 (+ 1467us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:07:10.965846 (+ 594us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:07:10.965849 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:07:10.965851 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:07:10.965933 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:07:10.966274 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:07:10.966277 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:07:10.966278 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:07:10.966453 (+ 175us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:07:10.966562 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0620 06:07:10.967059 (+ 497us) server_negotiation.cc:299] Negotiation successful
0620 06:07:10.967172 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.102.3@KRBTEST.COM" created.
I20260620 06:07:10.968240 17577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:07:10.954858 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.102.62:44137 (local address 127.11.102.2:37271)
0620 06:07:10.955313 (+ 455us) negotiation.cc:107] Waiting for socket to connect
0620 06:07:10.955355 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0620 06:07:10.956935 (+ 1580us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:07:10.957526 (+ 591us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:07:10.957533 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:07:10.957978 (+ 445us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:07:10.958657 (+ 679us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:07:10.958679 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:07:10.959569 (+ 890us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:07:10.959572 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:07:10.960510 (+ 938us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:07:10.960528 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:07:10.961061 (+ 533us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:07:10.961065 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:07:10.961247 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:07:10.961824 (+ 577us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:07:10.961839 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:07:10.963543 (+ 1704us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:07:10.965385 (+ 1842us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:07:10.965390 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:07:10.965405 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:07:10.965738 (+ 333us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:07:10.966032 (+ 294us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:07:10.966035 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:07:10.966037 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:07:10.966172 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:07:10.966562 (+ 390us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:07:10.966570 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:07:10.966810 (+ 240us) client_negotiation.cc:769] Sending connection context
0620 06:07:10.966995 (+ 185us) client_negotiation.cc:240] Negotiation successful
0620 06:07:10.967309 (+ 314us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":280,"thread_start_us":116,"threads_started":1}
I20260620 06:07:10.969554 17575 heartbeater.cc:347] Connected to a master server at 127.11.102.62:44137
I20260620 06:07:10.969794 17575 heartbeater.cc:464] Registering TS with master...
I20260620 06:07:10.970276 17575 heartbeater.cc:511] Master 127.11.102.62:44137 requested a full tablet report, sending...
I20260620 06:07:10.971267 17250 ts_manager.cc:194] Registered new tserver with Master: da49668cf7894333bc41bd9502fc0485 (127.11.102.2:37599)
I20260620 06:07:10.971905 17250 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.102.2@KRBTEST.COM'} at 127.11.102.2:37271
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.102.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781935603120653-11672-0/krb5kdc/kudu_127.11.102.3.keytab.
Entry for principal kudu/127.11.102.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781935603120653-11672-0/krb5kdc/kudu_127.11.102.3.keytab.
I20260620 06:07:10.982753 11672 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9HMp43/build/debug/bin/kudu
/tmp/dist-test-task9HMp43/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781935603120653-11672-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781935603120653-11672-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781935603120653-11672-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781935603120653-11672-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.102.3:0
--local_ip_for_outbound_sockets=127.11.102.3
--webserver_interface=127.11.102.3
--webserver_port=0
--tserver_master_addrs=127.11.102.62:44137
--keytab_file=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781935603120653-11672-0/krb5kdc/kudu_127.11.102.3.keytab
--principal=kudu/127.11.102.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.102.20:33757
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781935603120653-11672-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781935603120653-11672-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781935603120653-11672-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 06:07:11.090518 17580 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:07:11.090752 17580 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:07:11.090790 17580 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:07:11.094940 17580 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:07:11.094992 17580 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:07:11.095044 17580 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.102.3
I20260620 06:07:11.099828 17580 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.102.20:33757
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781935603120653-11672-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781935603120653-11672-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781935603120653-11672-0/krb5kdc/kudu_127.11.102.3.keytab
--principal=kudu/127.11.102.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.102.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781935603120653-11672-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.11.102.3
--webserver_port=0
--tserver_master_addrs=127.11.102.62:44137
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17580
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.102.3
--log_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781935603120653-11672-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 05:28:48 UTC on ccycloud.quasar-fixhms.root.comops.site
build id 4457020
I20260620 06:07:11.100688 17580 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:07:11.101642 17580 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 06:07:11.110692 17588 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:07:11.110728 17580 server_base.cc:1034] running on GCE node
W20260620 06:07:11.110807 17585 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:07:11.111043 17586 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:07:11.111388 17580 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:07:11.112035 17580 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:07:11.113220 17580 hybrid_clock.cc:648] HybridClock initialized: now 1781935631113212 us; error 29 us; skew 500 ppm
Jun 20 06:07:11 dist-test-slave-9hm5 krb5kdc[17205](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935631, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.102.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 06:07:11.115805 17580 init.cc:377] Logged in from keytab as kudu/127.11.102.3@KRBTEST.COM (short username kudu)
I20260620 06:07:11.116617 17580 webserver.cc:460] Webserver started at http://127.11.102.3:41539/ using document root <none> and password file <none>
I20260620 06:07:11.117168 17580 fs_manager.cc:359] Metadata directory not provided
I20260620 06:07:11.117259 17580 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:07:11.117520 17580 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 06:07:11.119397 17580 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781935603120653-11672-0/minicluster-data/ts-2/data/instance:
uuid: "f97dd06b0c3a4ba5814107fafca3ad9f"
format_stamp: "Formatted at 2026-06-20 06:07:11 on dist-test-slave-9hm5"
server_key: "a46057fdd001cc59504e11a9059a6093"
server_key_iv: "56d9c7f511af47e5620d63954959b3be"
server_key_version: "encryptionkey@0"
I20260620 06:07:11.119987 17580 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781935603120653-11672-0/minicluster-data/ts-2/wal/instance:
uuid: "f97dd06b0c3a4ba5814107fafca3ad9f"
format_stamp: "Formatted at 2026-06-20 06:07:11 on dist-test-slave-9hm5"
server_key: "a46057fdd001cc59504e11a9059a6093"
server_key_iv: "56d9c7f511af47e5620d63954959b3be"
server_key_version: "encryptionkey@0"
I20260620 06:07:11.123260 17580 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260620 06:07:11.125478 17595 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:07:11.126543 17580 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 06:07:11.126669 17580 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781935603120653-11672-0/minicluster-data/ts-2/data,/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781935603120653-11672-0/minicluster-data/ts-2/wal
uuid: "f97dd06b0c3a4ba5814107fafca3ad9f"
format_stamp: "Formatted at 2026-06-20 06:07:11 on dist-test-slave-9hm5"
server_key: "a46057fdd001cc59504e11a9059a6093"
server_key_iv: "56d9c7f511af47e5620d63954959b3be"
server_key_version: "encryptionkey@0"
I20260620 06:07:11.126770 17580 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781935603120653-11672-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781935603120653-11672-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781935603120653-11672-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:07:11.152817 17580 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:07:11.155481 17580 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:07:11.155694 17580 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:07:11.156229 17580 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 06:07:11.156989 17580 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 06:07:11.157058 17580 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 06:07:11.157123 17580 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 06:07:11.157159 17580 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 06:07:11.165540 17580 rpc_server.cc:307] RPC server started. Bound to: 127.11.102.3:39429
I20260620 06:07:11.165583 17708 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.102.3:39429 every 8 connection(s)
I20260620 06:07:11.166339 17580 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781935603120653-11672-0/minicluster-data/ts-2/data/info.pb
I20260620 06:07:11.168320 11672 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9HMp43/build/debug/bin/kudu as pid 17580
I20260620 06:07:11.168450 11672 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781935603120653-11672-0/minicluster-data/ts-2/wal/instance
I20260620 06:07:11.168715 11672 external_mini_cluster.cc:1428] Setting key 8e4a7dd7fa2be6737a643b832fb04ab9
Jun 20 06:07:11 dist-test-slave-9hm5 krb5kdc[17205](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935631, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.102.3@KRBTEST.COM for kudu/127.11.102.62@KRBTEST.COM
I20260620 06:07:11.179313 17289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:07:11.167988 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.102.3:36835 (local address 127.11.102.62:44137)
0620 06:07:11.168136 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0620 06:07:11.168141 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 06:07:11.168963 (+ 822us) server_negotiation.cc:407] Connection header received
0620 06:07:11.170318 (+ 1355us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:07:11.170321 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:07:11.170372 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:07:11.170458 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:07:11.171579 (+ 1121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:07:11.172156 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:07:11.173222 (+ 1066us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:07:11.173426 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:07:11.173503 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:07:11.175996 (+ 2493us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:07:11.175999 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:07:11.176001 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:07:11.176024 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:07:11.177547 (+ 1523us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:07:11.178018 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:07:11.178020 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:07:11.178021 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:07:11.178092 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:07:11.178399 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:07:11.178402 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:07:11.178404 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:07:11.178586 (+ 182us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:07:11.178675 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0620 06:07:11.179108 (+ 433us) server_negotiation.cc:299] Negotiation successful
0620 06:07:11.179200 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260620 06:07:11.180193 17711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:07:11.168304 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.102.62:44137 (local address 127.11.102.3:36835)
0620 06:07:11.168779 (+ 475us) negotiation.cc:107] Waiting for socket to connect
0620 06:07:11.168829 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0620 06:07:11.170133 (+ 1304us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:07:11.170647 (+ 514us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:07:11.170657 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:07:11.171017 (+ 360us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:07:11.171456 (+ 439us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:07:11.171468 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:07:11.172256 (+ 788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:07:11.172259 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:07:11.173119 (+ 860us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:07:11.173127 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:07:11.173545 (+ 418us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:07:11.173548 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:07:11.173675 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:07:11.174257 (+ 582us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:07:11.174273 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:07:11.175859 (+ 1586us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:07:11.177664 (+ 1805us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:07:11.177668 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:07:11.177679 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:07:11.177934 (+ 255us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:07:11.178167 (+ 233us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:07:11.178169 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:07:11.178171 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:07:11.178288 (+ 117us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:07:11.178685 (+ 397us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:07:11.178689 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:07:11.178910 (+ 221us) client_negotiation.cc:769] Sending connection context
0620 06:07:11.179057 (+ 147us) client_negotiation.cc:240] Negotiation successful
0620 06:07:11.179272 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":290,"thread_start_us":132,"threads_started":1}
I20260620 06:07:11.181284 17709 heartbeater.cc:347] Connected to a master server at 127.11.102.62:44137
I20260620 06:07:11.181528 17709 heartbeater.cc:464] Registering TS with master...
I20260620 06:07:11.182042 17709 heartbeater.cc:511] Master 127.11.102.62:44137 requested a full tablet report, sending...
I20260620 06:07:11.182971 17250 ts_manager.cc:194] Registered new tserver with Master: f97dd06b0c3a4ba5814107fafca3ad9f (127.11.102.3:39429)
I20260620 06:07:11.183521 17250 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.102.3@KRBTEST.COM'} at 127.11.102.3:36835
I20260620 06:07:11.193709 11672 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781935603120653-11672-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781935603120653-11672-0/krb5kdc/test-admin.keytab.
Jun 20 06:07:11 dist-test-slave-9hm5 krb5kdc[17205](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935631, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 06:07:11.208032 11672 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260620 06:07:11.210709 17716 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 06:07:11.209218 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51400 (local address 127.11.102.1:40349)
0620 06:07:11.209484 (+ 266us) server_negotiation.cc:206] Beginning negotiation
0620 06:07:11.209498 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 06:07:11.209522 (+ 24us) server_negotiation.cc:407] Connection header received
0620 06:07:11.209648 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:07:11.209653 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:07:11.209836 (+ 183us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:07:11.210003 (+ 167us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:07:11.210498 (+ 495us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51400: BlockingRecv error: recv got EOF from 127.0.0.1:51400 (error 108)
Metrics: {"server-negotiator.queue_time_us":173,"thread_start_us":70,"threads_started":1}
W20260620 06:07:11.210896 11672 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781935603120653-11672-0/krb5kdc/krb5cc' of type FILE
Jun 20 06:07:11 dist-test-slave-9hm5 krb5kdc[17205](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935631, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 06:07:11.211956 11672 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Jun 20 06:07:11 dist-test-slave-9hm5 krb5kdc[17205](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935631, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 06:07:11.212846 11672 init.cc:295] Successfully reacquired a new kerberos TGT
Jun 20 06:07:11 dist-test-slave-9hm5 krb5kdc[17205](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935631, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.102.1@KRBTEST.COM
I20260620 06:07:11.222510 17716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:07:11.213619 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51406 (local address 127.11.102.1:40349)
0620 06:07:11.213733 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0620 06:07:11.213737 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 06:07:11.213878 (+ 141us) server_negotiation.cc:407] Connection header received
0620 06:07:11.214019 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:07:11.214025 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:07:11.214067 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:07:11.214151 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:07:11.214660 (+ 509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:07:11.215420 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:07:11.216303 (+ 883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:07:11.216666 (+ 363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:07:11.216784 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:07:11.219116 (+ 2332us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:07:11.219124 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:07:11.219138 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:07:11.219160 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:07:11.220795 (+ 1635us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:07:11.221227 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:07:11.221232 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:07:11.221233 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:07:11.221314 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:07:11.221599 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:07:11.221602 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:07:11.221603 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:07:11.221835 (+ 232us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:07:11.222007 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0620 06:07:11.222189 (+ 182us) server_negotiation.cc:299] Negotiation successful
0620 06:07:11.222375 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
W20260620 06:07:11.226040 17716 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 06:07:11.225182 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51414 (local address 127.11.102.1:40349)
0620 06:07:11.225292 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0620 06:07:11.225295 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 06:07:11.225306 (+ 11us) server_negotiation.cc:407] Connection header received
0620 06:07:11.225398 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:07:11.225400 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:07:11.225429 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:07:11.225492 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:07:11.225940 (+ 448us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51414: BlockingRecv error: recv got EOF from 127.0.0.1:51414 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
W20260620 06:07:11.226073 11672 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781935603120653-11672-0/krb5kdc/krb5cc' of type FILE
Jun 20 06:07:11 dist-test-slave-9hm5 krb5kdc[17205](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935631, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 06:07:11.226967 11672 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Jun 20 06:07:11 dist-test-slave-9hm5 krb5kdc[17205](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935631, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 06:07:11.227849 11672 init.cc:295] Successfully reacquired a new kerberos TGT
Jun 20 06:07:11 dist-test-slave-9hm5 krb5kdc[17205](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935631, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.102.1@KRBTEST.COM
I20260620 06:07:11.236039 17716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:07:11.228572 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51426 (local address 127.11.102.1:40349)
0620 06:07:11.228675 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0620 06:07:11.228679 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 06:07:11.228772 (+ 93us) server_negotiation.cc:407] Connection header received
0620 06:07:11.228864 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:07:11.228866 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:07:11.228896 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:07:11.228955 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:07:11.229347 (+ 392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:07:11.229885 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:07:11.230724 (+ 839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:07:11.230950 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:07:11.231030 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:07:11.233215 (+ 2185us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:07:11.233218 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:07:11.233220 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:07:11.233239 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:07:11.234680 (+ 1441us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:07:11.235078 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:07:11.235080 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:07:11.235081 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:07:11.235146 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:07:11.235370 (+ 224us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:07:11.235372 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:07:11.235373 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:07:11.235508 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:07:11.235593 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0620 06:07:11.235863 (+ 270us) server_negotiation.cc:299] Negotiation successful
0620 06:07:11.235944 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
W20260620 06:07:11.238343 17716 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 06:07:11.237507 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51436 (local address 127.11.102.1:40349)
0620 06:07:11.237611 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0620 06:07:11.237614 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 06:07:11.237757 (+ 143us) server_negotiation.cc:407] Connection header received
0620 06:07:11.237830 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:07:11.237832 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:07:11.237881 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:07:11.237942 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:07:11.238246 (+ 304us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51436: BlockingRecv error: recv got EOF from 127.0.0.1:51436 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
W20260620 06:07:11.238484 11672 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781935603120653-11672-0/krb5kdc/krb5cc' of type FILE
Jun 20 06:07:11 dist-test-slave-9hm5 krb5kdc[17205](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935631, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 06:07:11.239481 11672 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Jun 20 06:07:11 dist-test-slave-9hm5 krb5kdc[17205](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935631, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 06:07:11.240449 11672 init.cc:295] Successfully reacquired a new kerberos TGT
Jun 20 06:07:11 dist-test-slave-9hm5 krb5kdc[17205](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935631, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.102.1@KRBTEST.COM
I20260620 06:07:11.248179 17716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:07:11.241147 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51452 (local address 127.11.102.1:40349)
0620 06:07:11.241245 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0620 06:07:11.241249 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 06:07:11.241377 (+ 128us) server_negotiation.cc:407] Connection header received
0620 06:07:11.241454 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:07:11.241456 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:07:11.241485 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:07:11.241548 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:07:11.242021 (+ 473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:07:11.242476 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:07:11.243273 (+ 797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:07:11.243479 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:07:11.243563 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:07:11.245596 (+ 2033us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:07:11.245599 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:07:11.245601 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:07:11.245619 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:07:11.246768 (+ 1149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:07:11.247149 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:07:11.247152 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:07:11.247153 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:07:11.247227 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:07:11.247471 (+ 244us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:07:11.247474 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:07:11.247475 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:07:11.247684 (+ 209us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:07:11.247811 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0620 06:07:11.247996 (+ 185us) server_negotiation.cc:299] Negotiation successful
0620 06:07:11.248083 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260620 06:07:11.249148 11672 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9HMp43/build/debug/bin/kudu with pid 17312
I20260620 06:07:11.255471 11672 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9HMp43/build/debug/bin/kudu with pid 17446
I20260620 06:07:11.260486 11672 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9HMp43/build/debug/bin/kudu with pid 17580
I20260620 06:07:11.265260 11672 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9HMp43/build/debug/bin/kudu with pid 17218
2026-06-20T06:07:11Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (2148 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935603120653-11672-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 06:07:11 dist-test-slave-9hm5 krb5kdc[17729](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 06:07:11 dist-test-slave-9hm5 krb5kdc[17729](info): set up 2 sockets
Jun 20 06:07:11 dist-test-slave-9hm5 krb5kdc[17729](info): commencing operation
krb5kdc: starting...
W20260620 06:07:12.292363 11672 mini_kdc.cc:121] Time spent starting KDC: real 1.006s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935603120653-11672-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935603120653-11672-0/krb5kdc/test-user.keytab.
Jun 20 06:07:12 dist-test-slave-9hm5 krb5kdc[17729](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935632, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T06:07:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T06:07:12Z Disabled control of system clock
WARNING: no policy specified for oryx/127.11.102.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.11.102.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.11.102.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935603120653-11672-0/krb5kdc/oryx_127.11.102.62.keytab.
Entry for principal oryx/127.11.102.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935603120653-11672-0/krb5kdc/oryx_127.11.102.62.keytab.
I20260620 06:07:12.410200 11672 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9HMp43/build/debug/bin/kudu
/tmp/dist-test-task9HMp43/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935603120653-11672-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935603120653-11672-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935603120653-11672-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935603120653-11672-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.102.62:44777
--webserver_interface=127.11.102.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935603120653-11672-0/krb5kdc/oryx_127.11.102.62.keytab
--principal=oryx/127.11.102.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.102.20:34945
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.102.62:44777
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935603120653-11672-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935603120653-11672-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935603120653-11672-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 06:07:12.518466 17742 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:07:12.518786 17742 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:07:12.518870 17742 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:07:12.522836 17742 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 06:07:12.522934 17742 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:07:12.522977 17742 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:07:12.523043 17742 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 06:07:12.523087 17742 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 06:07:12.528939 17742 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.102.20:34945
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935603120653-11672-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935603120653-11672-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.102.62:44777
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935603120653-11672-0/krb5kdc/oryx_127.11.102.62.keytab
--principal=oryx/127.11.102.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.102.62:44777
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935603120653-11672-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.102.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17742
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935603120653-11672-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 05:28:48 UTC on ccycloud.quasar-fixhms.root.comops.site
build id 4457020
I20260620 06:07:12.530023 17742 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:07:12.531064 17742 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 06:07:12.539588 17748 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:07:12.539672 17750 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:07:12.539597 17747 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:07:12.539911 17742 server_base.cc:1034] running on GCE node
I20260620 06:07:12.540445 17742 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:07:12.541414 17742 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:07:12.542572 17742 hybrid_clock.cc:648] HybridClock initialized: now 1781935632542554 us; error 29 us; skew 500 ppm
Jun 20 06:07:12 dist-test-slave-9hm5 krb5kdc[17729](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935632, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.102.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 06:07:12.548099 17742 init.cc:377] Logged in from keytab as oryx/127.11.102.62@KRBTEST.COM (short username oryx)
I20260620 06:07:12.548908 17742 webserver.cc:460] Webserver started at http://127.11.102.62:45333/ using document root <none> and password file <none>
I20260620 06:07:12.549412 17742 fs_manager.cc:359] Metadata directory not provided
I20260620 06:07:12.549471 17742 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:07:12.549629 17742 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 06:07:12.551558 17742 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935603120653-11672-0/minicluster-data/master-0/data/instance:
uuid: "d7d4500dc38a48a1858293d72d1fcfc2"
format_stamp: "Formatted at 2026-06-20 06:07:12 on dist-test-slave-9hm5"
server_key: "3bbddf7d4686a84c5e16791ef8f67de5"
server_key_iv: "68f5ffcfe5f5a747002cf9e7f0038043"
server_key_version: "encryptionkey@0"
I20260620 06:07:12.552129 17742 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935603120653-11672-0/minicluster-data/master-0/wal/instance:
uuid: "d7d4500dc38a48a1858293d72d1fcfc2"
format_stamp: "Formatted at 2026-06-20 06:07:12 on dist-test-slave-9hm5"
server_key: "3bbddf7d4686a84c5e16791ef8f67de5"
server_key_iv: "68f5ffcfe5f5a747002cf9e7f0038043"
server_key_version: "encryptionkey@0"
I20260620 06:07:12.555519 17742 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260620 06:07:12.557816 17757 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:07:12.558965 17742 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 06:07:12.559098 17742 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935603120653-11672-0/minicluster-data/master-0/data,/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935603120653-11672-0/minicluster-data/master-0/wal
uuid: "d7d4500dc38a48a1858293d72d1fcfc2"
format_stamp: "Formatted at 2026-06-20 06:07:12 on dist-test-slave-9hm5"
server_key: "3bbddf7d4686a84c5e16791ef8f67de5"
server_key_iv: "68f5ffcfe5f5a747002cf9e7f0038043"
server_key_version: "encryptionkey@0"
I20260620 06:07:12.559198 17742 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935603120653-11672-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935603120653-11672-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935603120653-11672-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:07:12.571923 17742 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:07:12.574640 17742 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:07:12.574836 17742 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:07:12.581887 17742 rpc_server.cc:307] RPC server started. Bound to: 127.11.102.62:44777
I20260620 06:07:12.582186 17809 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.102.62:44777 every 8 connection(s)
I20260620 06:07:12.582861 17742 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935603120653-11672-0/minicluster-data/master-0/data/info.pb
I20260620 06:07:12.585515 17810 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 06:07:12.586104 11672 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9HMp43/build/debug/bin/kudu as pid 17742
I20260620 06:07:12.586236 11672 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935603120653-11672-0/minicluster-data/master-0/wal/instance
I20260620 06:07:12.586551 11672 external_mini_cluster.cc:1428] Setting key 1197f5576cac8266743c5334d2dc57cf
I20260620 06:07:12.591260 17810 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d7d4500dc38a48a1858293d72d1fcfc2: Bootstrap starting.
I20260620 06:07:12.593355 17810 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d7d4500dc38a48a1858293d72d1fcfc2: Neither blocks nor log segments found. Creating new log.
I20260620 06:07:12.594027 17810 log.cc:824] T 00000000000000000000000000000000 P d7d4500dc38a48a1858293d72d1fcfc2: Log is configured to *not* fsync() on all Append() calls
Jun 20 06:07:12 dist-test-slave-9hm5 krb5kdc[17729](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935632, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.11.102.62@KRBTEST.COM
I20260620 06:07:12.595897 17810 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d7d4500dc38a48a1858293d72d1fcfc2: No bootstrap required, opened a new log
I20260620 06:07:12.599156 17810 raft_consensus.cc:348] T 00000000000000000000000000000000 P d7d4500dc38a48a1858293d72d1fcfc2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d7d4500dc38a48a1858293d72d1fcfc2" member_type: VOTER last_known_addr { host: "127.11.102.62" port: 44777 } }
I20260620 06:07:12.599450 17810 raft_consensus.cc:374] T 00000000000000000000000000000000 P d7d4500dc38a48a1858293d72d1fcfc2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 06:07:12.599534 17810 raft_consensus.cc:729] T 00000000000000000000000000000000 P d7d4500dc38a48a1858293d72d1fcfc2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d7d4500dc38a48a1858293d72d1fcfc2, State: Initialized, Role: FOLLOWER
I20260620 06:07:12.600008 17810 consensus_queue.cc:260] T 00000000000000000000000000000000 P d7d4500dc38a48a1858293d72d1fcfc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d7d4500dc38a48a1858293d72d1fcfc2" member_type: VOTER last_known_addr { host: "127.11.102.62" port: 44777 } }
I20260620 06:07:12.600126 17810 raft_consensus.cc:388] T 00000000000000000000000000000000 P d7d4500dc38a48a1858293d72d1fcfc2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 06:07:12.600215 17810 raft_consensus.cc:482] T 00000000000000000000000000000000 P d7d4500dc38a48a1858293d72d1fcfc2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 06:07:12.600322 17813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:07:12.587929 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54608 (local address 127.11.102.62:44777)
0620 06:07:12.588354 (+ 425us) server_negotiation.cc:206] Beginning negotiation
0620 06:07:12.588367 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 06:07:12.588400 (+ 33us) server_negotiation.cc:407] Connection header received
0620 06:07:12.589002 (+ 602us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:07:12.589018 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:07:12.589339 (+ 321us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:07:12.589664 (+ 325us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:07:12.590455 (+ 791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:07:12.591466 (+ 1011us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:07:12.592360 (+ 894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:07:12.592827 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:07:12.593004 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:07:12.595437 (+ 2433us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:07:12.595442 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:07:12.595454 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:07:12.595480 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:07:12.597174 (+ 1694us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:07:12.597643 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:07:12.597647 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:07:12.597652 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:07:12.597756 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:07:12.598043 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:07:12.598047 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:07:12.598048 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:07:12.598557 (+ 509us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:07:12.598781 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0620 06:07:12.599084 (+ 303us) server_negotiation.cc:299] Negotiation successful
0620 06:07:12.599361 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":293,"thread_start_us":120,"threads_started":1}
I20260620 06:07:12.600335 17810 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d7d4500dc38a48a1858293d72d1fcfc2 [term 0 FOLLOWER]: Advancing to term 1
I20260620 06:07:12.601225 17810 raft_consensus.cc:504] T 00000000000000000000000000000000 P d7d4500dc38a48a1858293d72d1fcfc2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d7d4500dc38a48a1858293d72d1fcfc2" member_type: VOTER last_known_addr { host: "127.11.102.62" port: 44777 } }
I20260620 06:07:12.601567 17810 leader_election.cc:302] T 00000000000000000000000000000000 P d7d4500dc38a48a1858293d72d1fcfc2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d7d4500dc38a48a1858293d72d1fcfc2; no voters:
I20260620 06:07:12.601888 17810 leader_election.cc:288] T 00000000000000000000000000000000 P d7d4500dc38a48a1858293d72d1fcfc2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 06:07:12.602003 17815 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d7d4500dc38a48a1858293d72d1fcfc2 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 06:07:12.602238 17815 raft_consensus.cc:686] T 00000000000000000000000000000000 P d7d4500dc38a48a1858293d72d1fcfc2 [term 1 LEADER]: Becoming Leader. State: Replica: d7d4500dc38a48a1858293d72d1fcfc2, State: Running, Role: LEADER
I20260620 06:07:12.602527 17815 consensus_queue.cc:237] T 00000000000000000000000000000000 P d7d4500dc38a48a1858293d72d1fcfc2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d7d4500dc38a48a1858293d72d1fcfc2" member_type: VOTER last_known_addr { host: "127.11.102.62" port: 44777 } }
I20260620 06:07:12.603027 17810 sys_catalog.cc:564] T 00000000000000000000000000000000 P d7d4500dc38a48a1858293d72d1fcfc2 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 06:07:12.604127 17817 sys_catalog.cc:455] T 00000000000000000000000000000000 P d7d4500dc38a48a1858293d72d1fcfc2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d7d4500dc38a48a1858293d72d1fcfc2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d7d4500dc38a48a1858293d72d1fcfc2" member_type: VOTER last_known_addr { host: "127.11.102.62" port: 44777 } } }
I20260620 06:07:12.604252 17817 sys_catalog.cc:458] T 00000000000000000000000000000000 P d7d4500dc38a48a1858293d72d1fcfc2 [sys.catalog]: This master's current role is: LEADER
I20260620 06:07:12.604528 17816 sys_catalog.cc:455] T 00000000000000000000000000000000 P d7d4500dc38a48a1858293d72d1fcfc2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d7d4500dc38a48a1858293d72d1fcfc2. Latest consensus state: current_term: 1 leader_uuid: "d7d4500dc38a48a1858293d72d1fcfc2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d7d4500dc38a48a1858293d72d1fcfc2" member_type: VOTER last_known_addr { host: "127.11.102.62" port: 44777 } } }
I20260620 06:07:12.604595 17824 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 06:07:12.604640 17816 sys_catalog.cc:458] T 00000000000000000000000000000000 P d7d4500dc38a48a1858293d72d1fcfc2 [sys.catalog]: This master's current role is: LEADER
I20260620 06:07:12.607525 17824 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 06:07:12.612950 17824 catalog_manager.cc:1347] Generated new cluster ID: bcc977c4637644dab45ae102d36c03ef
I20260620 06:07:12.613021 17824 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 06:07:12.624146 17824 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 06:07:12.624958 17824 catalog_manager.cc:1504] Loading token signing keys...
I20260620 06:07:12.631443 17824 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d7d4500dc38a48a1858293d72d1fcfc2: Generated new TSK 0
I20260620 06:07:12.632053 17824 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.11.102.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.11.102.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.11.102.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935603120653-11672-0/krb5kdc/oryx_127.11.102.1.keytab.
Entry for principal oryx/127.11.102.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935603120653-11672-0/krb5kdc/oryx_127.11.102.1.keytab.
I20260620 06:07:12.660774 11672 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9HMp43/build/debug/bin/kudu
/tmp/dist-test-task9HMp43/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935603120653-11672-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935603120653-11672-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935603120653-11672-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935603120653-11672-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.102.1:0
--local_ip_for_outbound_sockets=127.11.102.1
--webserver_interface=127.11.102.1
--webserver_port=0
--tserver_master_addrs=127.11.102.62:44777
--keytab_file=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935603120653-11672-0/krb5kdc/oryx_127.11.102.1.keytab
--principal=oryx/127.11.102.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.102.20:34945
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935603120653-11672-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935603120653-11672-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935603120653-11672-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 06:07:12.767737 17836 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:07:12.767989 17836 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:07:12.768064 17836 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:07:12.771986 17836 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:07:12.772068 17836 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:07:12.772153 17836 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.102.1
I20260620 06:07:12.777220 17836 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.102.20:34945
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935603120653-11672-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935603120653-11672-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935603120653-11672-0/krb5kdc/oryx_127.11.102.1.keytab
--principal=oryx/127.11.102.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.102.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935603120653-11672-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.11.102.1
--webserver_port=0
--tserver_master_addrs=127.11.102.62:44777
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17836
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.102.1
--log_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935603120653-11672-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 05:28:48 UTC on ccycloud.quasar-fixhms.root.comops.site
build id 4457020
I20260620 06:07:12.778347 17836 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:07:12.779560 17836 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 06:07:12.788606 17842 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:07:12.788614 17841 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:07:12.788627 17844 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:07:12.788975 17836 server_base.cc:1034] running on GCE node
I20260620 06:07:12.789381 17836 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:07:12.789991 17836 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:07:12.791175 17836 hybrid_clock.cc:648] HybridClock initialized: now 1781935632791164 us; error 36 us; skew 500 ppm
Jun 20 06:07:12 dist-test-slave-9hm5 krb5kdc[17729](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935632, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.102.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 06:07:12.793885 17836 init.cc:377] Logged in from keytab as oryx/127.11.102.1@KRBTEST.COM (short username oryx)
I20260620 06:07:12.794682 17836 webserver.cc:460] Webserver started at http://127.11.102.1:36757/ using document root <none> and password file <none>
I20260620 06:07:12.795215 17836 fs_manager.cc:359] Metadata directory not provided
I20260620 06:07:12.795303 17836 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:07:12.795523 17836 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 06:07:12.797544 17836 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935603120653-11672-0/minicluster-data/ts-0/data/instance:
uuid: "fb0783328ad3442fba39f260adf25308"
format_stamp: "Formatted at 2026-06-20 06:07:12 on dist-test-slave-9hm5"
server_key: "c256682846b1d9a11755b37708093731"
server_key_iv: "5e598148fb006f306cd6f4985ee7540d"
server_key_version: "encryptionkey@0"
I20260620 06:07:12.798094 17836 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935603120653-11672-0/minicluster-data/ts-0/wal/instance:
uuid: "fb0783328ad3442fba39f260adf25308"
format_stamp: "Formatted at 2026-06-20 06:07:12 on dist-test-slave-9hm5"
server_key: "c256682846b1d9a11755b37708093731"
server_key_iv: "5e598148fb006f306cd6f4985ee7540d"
server_key_version: "encryptionkey@0"
I20260620 06:07:12.801705 17836 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260620 06:07:12.804080 17851 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:07:12.805411 17836 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 06:07:12.805541 17836 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935603120653-11672-0/minicluster-data/ts-0/data,/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935603120653-11672-0/minicluster-data/ts-0/wal
uuid: "fb0783328ad3442fba39f260adf25308"
format_stamp: "Formatted at 2026-06-20 06:07:12 on dist-test-slave-9hm5"
server_key: "c256682846b1d9a11755b37708093731"
server_key_iv: "5e598148fb006f306cd6f4985ee7540d"
server_key_version: "encryptionkey@0"
I20260620 06:07:12.805644 17836 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935603120653-11672-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935603120653-11672-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935603120653-11672-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:07:12.817001 17836 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:07:12.820083 17836 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:07:12.820277 17836 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:07:12.820819 17836 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 06:07:12.821575 17836 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 06:07:12.821645 17836 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 06:07:12.821725 17836 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 06:07:12.821779 17836 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 06:07:12.830394 17836 rpc_server.cc:307] RPC server started. Bound to: 127.11.102.1:37681
I20260620 06:07:12.830430 17964 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.102.1:37681 every 8 connection(s)
I20260620 06:07:12.831214 17836 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935603120653-11672-0/minicluster-data/ts-0/data/info.pb
I20260620 06:07:12.836905 11672 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9HMp43/build/debug/bin/kudu as pid 17836
I20260620 06:07:12.837035 11672 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935603120653-11672-0/minicluster-data/ts-0/wal/instance
I20260620 06:07:12.837381 11672 external_mini_cluster.cc:1428] Setting key e87c42026c9bf38b3d7f995d22231d1b
Jun 20 06:07:12 dist-test-slave-9hm5 krb5kdc[17729](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935632, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.102.1@KRBTEST.COM for oryx/127.11.102.62@KRBTEST.COM
I20260620 06:07:12.845691 17813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:07:12.832874 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.102.1:47219 (local address 127.11.102.62:44777)
0620 06:07:12.833001 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0620 06:07:12.833005 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 06:07:12.833788 (+ 783us) server_negotiation.cc:407] Connection header received
0620 06:07:12.835095 (+ 1307us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:07:12.835099 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:07:12.835153 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:07:12.835233 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:07:12.836516 (+ 1283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:07:12.837362 (+ 846us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:07:12.838908 (+ 1546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:07:12.839262 (+ 354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:07:12.839399 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:07:12.841748 (+ 2349us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:07:12.841755 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:07:12.841761 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:07:12.841795 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:07:12.843569 (+ 1774us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:07:12.844082 (+ 513us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:07:12.844090 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:07:12.844094 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:07:12.844198 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:07:12.844519 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:07:12.844526 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:07:12.844531 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:07:12.844800 (+ 269us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:07:12.844926 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0620 06:07:12.845347 (+ 421us) server_negotiation.cc:299] Negotiation successful
0620 06:07:12.845478 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260620 06:07:12.846285 17967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:07:12.833234 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.102.62:44777 (local address 127.11.102.1:47219)
0620 06:07:12.833648 (+ 414us) negotiation.cc:107] Waiting for socket to connect
0620 06:07:12.833687 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0620 06:07:12.834898 (+ 1211us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:07:12.835392 (+ 494us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:07:12.835398 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:07:12.835744 (+ 346us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:07:12.836320 (+ 576us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:07:12.836342 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:07:12.837535 (+ 1193us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:07:12.837538 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:07:12.838779 (+ 1241us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:07:12.838790 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:07:12.839404 (+ 614us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:07:12.839406 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:07:12.839541 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:07:12.840196 (+ 655us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:07:12.840212 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:07:12.841578 (+ 1366us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:07:12.843721 (+ 2143us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:07:12.843725 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:07:12.843736 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:07:12.843968 (+ 232us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:07:12.844306 (+ 338us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:07:12.844308 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:07:12.844309 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:07:12.844425 (+ 116us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:07:12.844909 (+ 484us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:07:12.844913 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:07:12.845110 (+ 197us) client_negotiation.cc:769] Sending connection context
0620 06:07:12.845261 (+ 151us) client_negotiation.cc:240] Negotiation successful
0620 06:07:12.845478 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":244,"thread_start_us":101,"threads_started":1}
I20260620 06:07:12.847430 17965 heartbeater.cc:347] Connected to a master server at 127.11.102.62:44777
I20260620 06:07:12.847694 17965 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for oryx/127.11.102.2@KRBTEST.COM; defaulting to no policy
I20260620 06:07:12.848150 17965 heartbeater.cc:511] Master 127.11.102.62:44777 requested a full tablet report, sending...
I20260620 06:07:12.849576 17774 ts_manager.cc:194] Registered new tserver with Master: fb0783328ad3442fba39f260adf25308 (127.11.102.1:37681)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.11.102.2@KRBTEST.COM" created.
I20260620 06:07:12.850837 17774 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.11.102.1@KRBTEST.COM'} at 127.11.102.1:47219
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.11.102.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935603120653-11672-0/krb5kdc/oryx_127.11.102.2.keytab.
Entry for principal oryx/127.11.102.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935603120653-11672-0/krb5kdc/oryx_127.11.102.2.keytab.
I20260620 06:07:12.866055 11672 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9HMp43/build/debug/bin/kudu
/tmp/dist-test-task9HMp43/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935603120653-11672-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935603120653-11672-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935603120653-11672-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935603120653-11672-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.102.2:0
--local_ip_for_outbound_sockets=127.11.102.2
--webserver_interface=127.11.102.2
--webserver_port=0
--tserver_master_addrs=127.11.102.62:44777
--keytab_file=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935603120653-11672-0/krb5kdc/oryx_127.11.102.2.keytab
--principal=oryx/127.11.102.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.102.20:34945
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935603120653-11672-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935603120653-11672-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935603120653-11672-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 06:07:12.973929 17970 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:07:12.974164 17970 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:07:12.974228 17970 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:07:12.978092 17970 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:07:12.978145 17970 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:07:12.978200 17970 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.102.2
I20260620 06:07:12.983273 17970 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.102.20:34945
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935603120653-11672-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935603120653-11672-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935603120653-11672-0/krb5kdc/oryx_127.11.102.2.keytab
--principal=oryx/127.11.102.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.102.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935603120653-11672-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.11.102.2
--webserver_port=0
--tserver_master_addrs=127.11.102.62:44777
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17970
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.102.2
--log_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935603120653-11672-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 05:28:48 UTC on ccycloud.quasar-fixhms.root.comops.site
build id 4457020
I20260620 06:07:12.984256 17970 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:07:12.985235 17970 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 06:07:12.994415 17976 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:07:12.994422 17975 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:07:12.994432 17978 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:07:12.994490 17970 server_base.cc:1034] running on GCE node
I20260620 06:07:12.994966 17970 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:07:12.995662 17970 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:07:12.996833 17970 hybrid_clock.cc:648] HybridClock initialized: now 1781935632996788 us; error 64 us; skew 500 ppm
Jun 20 06:07:12 dist-test-slave-9hm5 krb5kdc[17729](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935632, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.102.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 06:07:12.999539 17970 init.cc:377] Logged in from keytab as oryx/127.11.102.2@KRBTEST.COM (short username oryx)
I20260620 06:07:13.000378 17970 webserver.cc:460] Webserver started at http://127.11.102.2:34063/ using document root <none> and password file <none>
I20260620 06:07:13.000900 17970 fs_manager.cc:359] Metadata directory not provided
I20260620 06:07:13.000955 17970 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:07:13.001156 17970 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 06:07:13.003162 17970 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935603120653-11672-0/minicluster-data/ts-1/data/instance:
uuid: "f569ba0a0ea74d078dabe54eb42b9e21"
format_stamp: "Formatted at 2026-06-20 06:07:12 on dist-test-slave-9hm5"
server_key: "8bc93c39640867ba8b46e2563ae2aa8e"
server_key_iv: "b8ea8111e3e5e863cbea9878ae36dd87"
server_key_version: "encryptionkey@0"
I20260620 06:07:13.003835 17970 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935603120653-11672-0/minicluster-data/ts-1/wal/instance:
uuid: "f569ba0a0ea74d078dabe54eb42b9e21"
format_stamp: "Formatted at 2026-06-20 06:07:12 on dist-test-slave-9hm5"
server_key: "8bc93c39640867ba8b46e2563ae2aa8e"
server_key_iv: "b8ea8111e3e5e863cbea9878ae36dd87"
server_key_version: "encryptionkey@0"
I20260620 06:07:13.007524 17970 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260620 06:07:13.010012 17985 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:07:13.011160 17970 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 06:07:13.011284 17970 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935603120653-11672-0/minicluster-data/ts-1/data,/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935603120653-11672-0/minicluster-data/ts-1/wal
uuid: "f569ba0a0ea74d078dabe54eb42b9e21"
format_stamp: "Formatted at 2026-06-20 06:07:12 on dist-test-slave-9hm5"
server_key: "8bc93c39640867ba8b46e2563ae2aa8e"
server_key_iv: "b8ea8111e3e5e863cbea9878ae36dd87"
server_key_version: "encryptionkey@0"
I20260620 06:07:13.011391 17970 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935603120653-11672-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935603120653-11672-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935603120653-11672-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:07:13.042730 17970 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:07:13.046173 17970 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:07:13.046373 17970 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:07:13.046959 17970 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 06:07:13.047814 17970 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 06:07:13.047879 17970 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 06:07:13.047947 17970 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 06:07:13.047994 17970 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 06:07:13.056373 17970 rpc_server.cc:307] RPC server started. Bound to: 127.11.102.2:33643
I20260620 06:07:13.056393 18098 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.102.2:33643 every 8 connection(s)
I20260620 06:07:13.057230 17970 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935603120653-11672-0/minicluster-data/ts-1/data/info.pb
I20260620 06:07:13.061945 11672 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9HMp43/build/debug/bin/kudu as pid 17970
I20260620 06:07:13.062070 11672 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935603120653-11672-0/minicluster-data/ts-1/wal/instance
I20260620 06:07:13.062435 11672 external_mini_cluster.cc:1428] Setting key a1e316134e224d90a16cc87c10c880a4
Jun 20 06:07:13 dist-test-slave-9hm5 krb5kdc[17729](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935632, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.102.2@KRBTEST.COM for oryx/127.11.102.62@KRBTEST.COM
I20260620 06:07:13.071439 17813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:07:13.058983 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.102.2:52325 (local address 127.11.102.62:44777)
0620 06:07:13.059141 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0620 06:07:13.059144 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 06:07:13.059960 (+ 816us) server_negotiation.cc:407] Connection header received
0620 06:07:13.061314 (+ 1354us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:07:13.061318 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:07:13.061369 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:07:13.061474 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:07:13.062904 (+ 1430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:07:13.063549 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:07:13.064713 (+ 1164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:07:13.065016 (+ 303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:07:13.065127 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:07:13.068000 (+ 2873us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:07:13.068004 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:07:13.068006 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:07:13.068028 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:07:13.069476 (+ 1448us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:07:13.069975 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:07:13.069980 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:07:13.069989 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:07:13.070061 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:07:13.070388 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:07:13.070394 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:07:13.070396 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:07:13.070576 (+ 180us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:07:13.070692 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0620 06:07:13.071164 (+ 472us) server_negotiation.cc:299] Negotiation successful
0620 06:07:13.071278 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
WARNING: no policy specified for oryx/127.11.102.3@KRBTEST.COM; defaulting to no policy
I20260620 06:07:13.072252 18101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:07:13.059335 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.102.62:44777 (local address 127.11.102.2:52325)
0620 06:07:13.059808 (+ 473us) negotiation.cc:107] Waiting for socket to connect
0620 06:07:13.059850 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0620 06:07:13.061115 (+ 1265us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:07:13.061646 (+ 531us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:07:13.061653 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:07:13.062054 (+ 401us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:07:13.062673 (+ 619us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:07:13.062696 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:07:13.063687 (+ 991us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:07:13.063691 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:07:13.064589 (+ 898us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:07:13.064597 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:07:13.065161 (+ 564us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:07:13.065165 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:07:13.065395 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:07:13.066023 (+ 628us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:07:13.066051 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:07:13.067830 (+ 1779us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:07:13.069597 (+ 1767us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:07:13.069603 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:07:13.069617 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:07:13.069863 (+ 246us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:07:13.070152 (+ 289us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:07:13.070156 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:07:13.070158 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:07:13.070296 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:07:13.070676 (+ 380us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:07:13.070693 (+ 17us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:07:13.070933 (+ 240us) client_negotiation.cc:769] Sending connection context
0620 06:07:13.071125 (+ 192us) client_negotiation.cc:240] Negotiation successful
0620 06:07:13.071427 (+ 302us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":295,"thread_start_us":118,"threads_started":1}
I20260620 06:07:13.073385 18099 heartbeater.cc:347] Connected to a master server at 127.11.102.62:44777
I20260620 06:07:13.073740 18099 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.11.102.3@KRBTEST.COM" created.
I20260620 06:07:13.074282 18099 heartbeater.cc:511] Master 127.11.102.62:44777 requested a full tablet report, sending...
I20260620 06:07:13.075342 17774 ts_manager.cc:194] Registered new tserver with Master: f569ba0a0ea74d078dabe54eb42b9e21 (127.11.102.2:33643)
I20260620 06:07:13.076145 17774 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.11.102.2@KRBTEST.COM'} at 127.11.102.2:52325
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.11.102.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935603120653-11672-0/krb5kdc/oryx_127.11.102.3.keytab.
Entry for principal oryx/127.11.102.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935603120653-11672-0/krb5kdc/oryx_127.11.102.3.keytab.
I20260620 06:07:13.089955 11672 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9HMp43/build/debug/bin/kudu
/tmp/dist-test-task9HMp43/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935603120653-11672-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935603120653-11672-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935603120653-11672-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935603120653-11672-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.102.3:0
--local_ip_for_outbound_sockets=127.11.102.3
--webserver_interface=127.11.102.3
--webserver_port=0
--tserver_master_addrs=127.11.102.62:44777
--keytab_file=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935603120653-11672-0/krb5kdc/oryx_127.11.102.3.keytab
--principal=oryx/127.11.102.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.102.20:34945
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935603120653-11672-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935603120653-11672-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935603120653-11672-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 06:07:13.197947 18104 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:07:13.198235 18104 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:07:13.198312 18104 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:07:13.202374 18104 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:07:13.202452 18104 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:07:13.202539 18104 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.102.3
I20260620 06:07:13.207928 18104 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.102.20:34945
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935603120653-11672-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935603120653-11672-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935603120653-11672-0/krb5kdc/oryx_127.11.102.3.keytab
--principal=oryx/127.11.102.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.102.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935603120653-11672-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.11.102.3
--webserver_port=0
--tserver_master_addrs=127.11.102.62:44777
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18104
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.102.3
--log_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935603120653-11672-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 05:28:48 UTC on ccycloud.quasar-fixhms.root.comops.site
build id 4457020
I20260620 06:07:13.209070 18104 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:07:13.210094 18104 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 06:07:13.219234 18112 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:07:13.219269 18109 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:07:13.219281 18110 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:07:13.219435 18104 server_base.cc:1034] running on GCE node
I20260620 06:07:13.219894 18104 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:07:13.220435 18104 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:07:13.221590 18104 hybrid_clock.cc:648] HybridClock initialized: now 1781935633221578 us; error 24 us; skew 500 ppm
Jun 20 06:07:13 dist-test-slave-9hm5 krb5kdc[17729](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935633, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.102.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 06:07:13.224159 18104 init.cc:377] Logged in from keytab as oryx/127.11.102.3@KRBTEST.COM (short username oryx)
I20260620 06:07:13.224921 18104 webserver.cc:460] Webserver started at http://127.11.102.3:43289/ using document root <none> and password file <none>
I20260620 06:07:13.225447 18104 fs_manager.cc:359] Metadata directory not provided
I20260620 06:07:13.225534 18104 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:07:13.225754 18104 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 06:07:13.227695 18104 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935603120653-11672-0/minicluster-data/ts-2/data/instance:
uuid: "d4b3fbe7b8664c968f75cc26b72410d7"
format_stamp: "Formatted at 2026-06-20 06:07:13 on dist-test-slave-9hm5"
server_key: "fd44a5564a95678a66d44c25afe1a9e8"
server_key_iv: "1faf2a938ebbaba58ec523f52a54b31f"
server_key_version: "encryptionkey@0"
I20260620 06:07:13.228240 18104 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935603120653-11672-0/minicluster-data/ts-2/wal/instance:
uuid: "d4b3fbe7b8664c968f75cc26b72410d7"
format_stamp: "Formatted at 2026-06-20 06:07:13 on dist-test-slave-9hm5"
server_key: "fd44a5564a95678a66d44c25afe1a9e8"
server_key_iv: "1faf2a938ebbaba58ec523f52a54b31f"
server_key_version: "encryptionkey@0"
I20260620 06:07:13.231581 18104 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.002s
I20260620 06:07:13.233806 18119 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:07:13.234913 18104 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 06:07:13.235046 18104 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935603120653-11672-0/minicluster-data/ts-2/data,/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935603120653-11672-0/minicluster-data/ts-2/wal
uuid: "d4b3fbe7b8664c968f75cc26b72410d7"
format_stamp: "Formatted at 2026-06-20 06:07:13 on dist-test-slave-9hm5"
server_key: "fd44a5564a95678a66d44c25afe1a9e8"
server_key_iv: "1faf2a938ebbaba58ec523f52a54b31f"
server_key_version: "encryptionkey@0"
I20260620 06:07:13.235146 18104 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935603120653-11672-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935603120653-11672-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935603120653-11672-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:07:13.266738 18104 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:07:13.269817 18104 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:07:13.270025 18104 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:07:13.270627 18104 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 06:07:13.271430 18104 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 06:07:13.271500 18104 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 06:07:13.271571 18104 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 06:07:13.271615 18104 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 06:07:13.280586 18104 rpc_server.cc:307] RPC server started. Bound to: 127.11.102.3:36875
I20260620 06:07:13.280615 18232 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.102.3:36875 every 8 connection(s)
I20260620 06:07:13.281453 18104 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935603120653-11672-0/minicluster-data/ts-2/data/info.pb
I20260620 06:07:13.285782 11672 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9HMp43/build/debug/bin/kudu as pid 18104
I20260620 06:07:13.285902 11672 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935603120653-11672-0/minicluster-data/ts-2/wal/instance
I20260620 06:07:13.286175 11672 external_mini_cluster.cc:1428] Setting key d76e8f7c60bf4da04cfe660f85cb83c2
Jun 20 06:07:13 dist-test-slave-9hm5 krb5kdc[17729](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935633, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.102.3@KRBTEST.COM for oryx/127.11.102.62@KRBTEST.COM
I20260620 06:07:13.294718 17813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:07:13.283346 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.102.3:47209 (local address 127.11.102.62:44777)
0620 06:07:13.283485 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0620 06:07:13.283489 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 06:07:13.284378 (+ 889us) server_negotiation.cc:407] Connection header received
0620 06:07:13.285655 (+ 1277us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:07:13.285659 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:07:13.285720 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:07:13.285820 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:07:13.287338 (+ 1518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:07:13.287912 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:07:13.289186 (+ 1274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:07:13.289394 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:07:13.289473 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:07:13.291767 (+ 2294us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:07:13.291770 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:07:13.291772 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:07:13.291793 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:07:13.293014 (+ 1221us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:07:13.293441 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:07:13.293443 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:07:13.293444 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:07:13.293518 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:07:13.293790 (+ 272us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:07:13.293792 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:07:13.293793 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:07:13.293966 (+ 173us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:07:13.294038 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0620 06:07:13.294488 (+ 450us) server_negotiation.cc:299] Negotiation successful
0620 06:07:13.294598 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260620 06:07:13.295486 18235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:07:13.283799 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.102.62:44777 (local address 127.11.102.3:47209)
0620 06:07:13.284228 (+ 429us) negotiation.cc:107] Waiting for socket to connect
0620 06:07:13.284268 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0620 06:07:13.285453 (+ 1185us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:07:13.286067 (+ 614us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:07:13.286074 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:07:13.286497 (+ 423us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:07:13.287191 (+ 694us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:07:13.287207 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:07:13.288043 (+ 836us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:07:13.288046 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:07:13.289081 (+ 1035us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:07:13.289090 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:07:13.289516 (+ 426us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:07:13.289518 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:07:13.289648 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:07:13.290166 (+ 518us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:07:13.290182 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:07:13.291628 (+ 1446us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:07:13.293108 (+ 1480us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:07:13.293113 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:07:13.293124 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:07:13.293359 (+ 235us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:07:13.293594 (+ 235us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:07:13.293596 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:07:13.293597 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:07:13.293717 (+ 120us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:07:13.294048 (+ 331us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:07:13.294053 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:07:13.294256 (+ 203us) client_negotiation.cc:769] Sending connection context
0620 06:07:13.294402 (+ 146us) client_negotiation.cc:240] Negotiation successful
0620 06:07:13.294628 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":254,"thread_start_us":106,"threads_started":1}
I20260620 06:07:13.296588 18233 heartbeater.cc:347] Connected to a master server at 127.11.102.62:44777
I20260620 06:07:13.296844 18233 heartbeater.cc:464] Registering TS with master...
I20260620 06:07:13.297333 18233 heartbeater.cc:511] Master 127.11.102.62:44777 requested a full tablet report, sending...
I20260620 06:07:13.298326 17774 ts_manager.cc:194] Registered new tserver with Master: d4b3fbe7b8664c968f75cc26b72410d7 (127.11.102.3:36875)
I20260620 06:07:13.299101 17774 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.11.102.3@KRBTEST.COM'} at 127.11.102.3:47209
I20260620 06:07:13.300151 11672 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.11.102.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.11.102.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.11.102.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935603120653-11672-0/krb5kdc/oryx_127.11.102.61.keytab.
Entry for principal oryx/127.11.102.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935603120653-11672-0/krb5kdc/oryx_127.11.102.61.keytab.
I20260620 06:07:13.326141 11672 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9HMp43/build/debug/bin/kudu
/tmp/dist-test-task9HMp43/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935603120653-11672-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935603120653-11672-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935603120653-11672-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935603120653-11672-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.102.61:39977
--webserver_interface=127.11.102.61
--webserver_port=0
--keytab_file=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935603120653-11672-0/krb5kdc/oryx_127.11.102.61.keytab
--principal=oryx/127.11.102.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.102.20:34945
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.102.62:44777,127.11.102.61:39977
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935603120653-11672-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935603120653-11672-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935603120653-11672-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 06:07:13.432957 18238 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:07:13.433188 18238 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:07:13.433228 18238 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:07:13.437150 18238 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 06:07:13.437199 18238 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:07:13.437218 18238 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:07:13.437234 18238 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 06:07:13.437273 18238 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 06:07:13.442521 18238 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.102.20:34945
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935603120653-11672-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935603120653-11672-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.11.102.62:44777,127.11.102.61:39977
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935603120653-11672-0/krb5kdc/oryx_127.11.102.61.keytab
--principal=oryx/127.11.102.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.102.61:39977
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935603120653-11672-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.11.102.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18238
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935603120653-11672-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 05:28:48 UTC on ccycloud.quasar-fixhms.root.comops.site
build id 4457020
W20260620 06:07:13.442723 18238 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.11.102.62:44777,127.11.102.61:39977'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260620 06:07:13.443512 18238 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:07:13.444484 18238 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 06:07:13.452354 18246 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:07:13.452474 18244 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:07:13.452411 18243 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:07:13.452550 18238 server_base.cc:1034] running on GCE node
I20260620 06:07:13.453135 18238 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:07:13.454166 18238 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:07:13.455348 18238 hybrid_clock.cc:648] HybridClock initialized: now 1781935633455331 us; error 35 us; skew 500 ppm
Jun 20 06:07:13 dist-test-slave-9hm5 krb5kdc[17729](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935633, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.102.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 06:07:13.458031 18238 init.cc:377] Logged in from keytab as oryx/127.11.102.61@KRBTEST.COM (short username oryx)
I20260620 06:07:13.458817 18238 webserver.cc:460] Webserver started at http://127.11.102.61:40839/ using document root <none> and password file <none>
I20260620 06:07:13.459359 18238 fs_manager.cc:359] Metadata directory not provided
I20260620 06:07:13.459450 18238 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:07:13.459720 18238 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 06:07:13.461619 18238 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935603120653-11672-0/minicluster-data/master-1/data/instance:
uuid: "0a9e7156fd5c4d399486dc5932b407b2"
format_stamp: "Formatted at 2026-06-20 06:07:13 on dist-test-slave-9hm5"
server_key: "02d4284fc9e50f19db1fb4050a8eebb4"
server_key_iv: "182e2cca2183c69f2163b6f3e9330b58"
server_key_version: "encryptionkey@0"
I20260620 06:07:13.462177 18238 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935603120653-11672-0/minicluster-data/master-1/wal/instance:
uuid: "0a9e7156fd5c4d399486dc5932b407b2"
format_stamp: "Formatted at 2026-06-20 06:07:13 on dist-test-slave-9hm5"
server_key: "02d4284fc9e50f19db1fb4050a8eebb4"
server_key_iv: "182e2cca2183c69f2163b6f3e9330b58"
server_key_version: "encryptionkey@0"
I20260620 06:07:13.465736 18238 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260620 06:07:13.467990 18253 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:07:13.469115 18238 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 06:07:13.469254 18238 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935603120653-11672-0/minicluster-data/master-1/data,/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935603120653-11672-0/minicluster-data/master-1/wal
uuid: "0a9e7156fd5c4d399486dc5932b407b2"
format_stamp: "Formatted at 2026-06-20 06:07:13 on dist-test-slave-9hm5"
server_key: "02d4284fc9e50f19db1fb4050a8eebb4"
server_key_iv: "182e2cca2183c69f2163b6f3e9330b58"
server_key_version: "encryptionkey@0"
I20260620 06:07:13.469358 18238 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935603120653-11672-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935603120653-11672-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935603120653-11672-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:07:13.485623 18238 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:07:13.488277 18238 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:07:13.488468 18238 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:07:13.495098 18238 rpc_server.cc:307] RPC server started. Bound to: 127.11.102.61:39977
I20260620 06:07:13.495393 18305 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.102.61:39977 every 8 connection(s)
I20260620 06:07:13.496083 18238 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935603120653-11672-0/minicluster-data/master-1/data/info.pb
I20260620 06:07:13.499701 18306 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 06:07:13.501952 11672 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9HMp43/build/debug/bin/kudu as pid 18238
I20260620 06:07:13.504972 18306 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.11.102.62" port: 44777 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 06:07:13 dist-test-slave-9hm5 krb5kdc[17729](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935632, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.11.102.61@KRBTEST.COM
Jun 20 06:07:13 dist-test-slave-9hm5 krb5kdc[17729](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935633, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.102.61@KRBTEST.COM for oryx/127.11.102.62@KRBTEST.COM
I20260620 06:07:13.516665 17813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:07:13.507174 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54614 (local address 127.11.102.62:44777)
0620 06:07:13.507316 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0620 06:07:13.507320 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 06:07:13.507658 (+ 338us) server_negotiation.cc:407] Connection header received
0620 06:07:13.507769 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:07:13.507772 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:07:13.507819 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:07:13.507896 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:07:13.508660 (+ 764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:07:13.509165 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:07:13.510175 (+ 1010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:07:13.510398 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:07:13.510522 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:07:13.513157 (+ 2635us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:07:13.513161 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:07:13.513163 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:07:13.513195 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:07:13.514702 (+ 1507us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:07:13.515201 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:07:13.515207 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:07:13.515212 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:07:13.515298 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:07:13.515641 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:07:13.515644 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:07:13.515646 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:07:13.515883 (+ 237us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:07:13.515990 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0620 06:07:13.516353 (+ 363us) server_negotiation.cc:299] Negotiation successful
0620 06:07:13.516465 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260620 06:07:13.517373 18310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:07:13.507136 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.102.62:44777 (local address 127.0.0.1:54614)
0620 06:07:13.507507 (+ 371us) negotiation.cc:107] Waiting for socket to connect
0620 06:07:13.507553 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0620 06:07:13.507689 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:07:13.507961 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:07:13.507966 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:07:13.508375 (+ 409us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:07:13.508546 (+ 171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:07:13.508553 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:07:13.509289 (+ 736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:07:13.509293 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:07:13.510055 (+ 762us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:07:13.510064 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:07:13.510551 (+ 487us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:07:13.510554 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:07:13.510661 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:07:13.511171 (+ 510us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:07:13.511185 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:07:13.513028 (+ 1843us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:07:13.514831 (+ 1803us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:07:13.514840 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:07:13.514849 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:07:13.515108 (+ 259us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:07:13.515398 (+ 290us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:07:13.515404 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:07:13.515409 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:07:13.515520 (+ 111us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:07:13.515981 (+ 461us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:07:13.515986 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:07:13.516134 (+ 148us) client_negotiation.cc:769] Sending connection context
0620 06:07:13.516291 (+ 157us) client_negotiation.cc:240] Negotiation successful
0620 06:07:13.516610 (+ 319us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":215,"mutex_wait_us":3,"thread_start_us":112,"threads_started":1}
I20260620 06:07:13.517372 18309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:07:13.503728 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52448 (local address 127.11.102.61:39977)
0620 06:07:13.504557 (+ 829us) server_negotiation.cc:206] Beginning negotiation
0620 06:07:13.504574 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0620 06:07:13.504613 (+ 39us) server_negotiation.cc:407] Connection header received
0620 06:07:13.505421 (+ 808us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:07:13.505443 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:07:13.505845 (+ 402us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:07:13.506174 (+ 329us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:07:13.506890 (+ 716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:07:13.508339 (+ 1449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:07:13.509232 (+ 893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:07:13.509824 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:07:13.509998 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:07:13.512599 (+ 2601us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:07:13.512606 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:07:13.512620 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:07:13.512652 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:07:13.514468 (+ 1816us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:07:13.515010 (+ 542us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:07:13.515015 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:07:13.515017 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:07:13.515127 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:07:13.515442 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:07:13.515445 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:07:13.515446 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:07:13.515961 (+ 515us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:07:13.516133 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0620 06:07:13.516353 (+ 220us) server_negotiation.cc:299] Negotiation successful
0620 06:07:13.516677 (+ 324us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":27,"server-negotiator.queue_time_us":693,"thread_start_us":156,"threads_started":1}
I20260620 06:07:13.518627 18306 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.11.102.61" port: 39977 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 06:07:13 dist-test-slave-9hm5 krb5kdc[17729](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935633, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.102.61@KRBTEST.COM for oryx/127.11.102.61@KRBTEST.COM
I20260620 06:07:13.525579 18310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:07:13.519036 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.102.61:39977 (local address 127.0.0.1:52450)
0620 06:07:13.519141 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0620 06:07:13.519154 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 06:07:13.519220 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:07:13.519497 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:07:13.519504 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:07:13.519686 (+ 182us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:07:13.519768 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:07:13.519773 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:07:13.520747 (+ 974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:07:13.520749 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:07:13.521226 (+ 477us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:07:13.521233 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:07:13.521619 (+ 386us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:07:13.521621 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:07:13.521685 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:07:13.522013 (+ 328us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:07:13.522023 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:07:13.523071 (+ 1048us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:07:13.524513 (+ 1442us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:07:13.524515 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:07:13.524516 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:07:13.524686 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:07:13.524916 (+ 230us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:07:13.524918 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:07:13.524919 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:07:13.524985 (+ 66us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:07:13.525257 (+ 272us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:07:13.525258 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:07:13.525319 (+ 61us) client_negotiation.cc:769] Sending connection context
0620 06:07:13.525390 (+ 71us) client_negotiation.cc:240] Negotiation successful
0620 06:07:13.525468 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":47}
I20260620 06:07:13.525635 18309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:07:13.519112 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52450 (local address 127.11.102.61:39977)
0620 06:07:13.519198 (+ 86us) server_negotiation.cc:206] Beginning negotiation
0620 06:07:13.519208 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 06:07:13.519218 (+ 10us) server_negotiation.cc:407] Connection header received
0620 06:07:13.519316 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:07:13.519319 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:07:13.519353 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:07:13.519412 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:07:13.519867 (+ 455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:07:13.520659 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:07:13.521314 (+ 655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:07:13.521534 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:07:13.521605 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:07:13.523180 (+ 1575us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:07:13.523183 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:07:13.523185 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:07:13.523204 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:07:13.524423 (+ 1219us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:07:13.524762 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:07:13.524764 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:07:13.524765 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:07:13.524827 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:07:13.525055 (+ 228us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:07:13.525057 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:07:13.525058 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:07:13.525185 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:07:13.525252 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0620 06:07:13.525439 (+ 187us) server_negotiation.cc:299] Negotiation successful
0620 06:07:13.525512 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260620 06:07:13.528075 18306 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0a9e7156fd5c4d399486dc5932b407b2: Bootstrap starting.
I20260620 06:07:13.530125 18306 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0a9e7156fd5c4d399486dc5932b407b2: Neither blocks nor log segments found. Creating new log.
I20260620 06:07:13.530804 18306 log.cc:824] T 00000000000000000000000000000000 P 0a9e7156fd5c4d399486dc5932b407b2: Log is configured to *not* fsync() on all Append() calls
I20260620 06:07:13.532588 18306 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0a9e7156fd5c4d399486dc5932b407b2: No bootstrap required, opened a new log
I20260620 06:07:13.535219 18306 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0a9e7156fd5c4d399486dc5932b407b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7d4500dc38a48a1858293d72d1fcfc2" member_type: VOTER last_known_addr { host: "127.11.102.62" port: 44777 } } peers { permanent_uuid: "0a9e7156fd5c4d399486dc5932b407b2" member_type: VOTER last_known_addr { host: "127.11.102.61" port: 39977 } }
I20260620 06:07:13.535430 18306 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0a9e7156fd5c4d399486dc5932b407b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 06:07:13.535516 18306 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0a9e7156fd5c4d399486dc5932b407b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a9e7156fd5c4d399486dc5932b407b2, State: Initialized, Role: FOLLOWER
I20260620 06:07:13.535959 18306 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0a9e7156fd5c4d399486dc5932b407b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7d4500dc38a48a1858293d72d1fcfc2" member_type: VOTER last_known_addr { host: "127.11.102.62" port: 44777 } } peers { permanent_uuid: "0a9e7156fd5c4d399486dc5932b407b2" member_type: VOTER last_known_addr { host: "127.11.102.61" port: 39977 } }
I20260620 06:07:13.536375 18312 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0a9e7156fd5c4d399486dc5932b407b2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7d4500dc38a48a1858293d72d1fcfc2" member_type: VOTER last_known_addr { host: "127.11.102.62" port: 44777 } } peers { permanent_uuid: "0a9e7156fd5c4d399486dc5932b407b2" member_type: VOTER last_known_addr { host: "127.11.102.61" port: 39977 } } }
I20260620 06:07:13.536486 18312 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0a9e7156fd5c4d399486dc5932b407b2 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 06:07:13.537007 18306 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0a9e7156fd5c4d399486dc5932b407b2 [sys.catalog]: configured and running, proceeding with master startup.
W20260620 06:07:13.541556 18323 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 0a9e7156fd5c4d399486dc5932b407b2: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 06:07:13.541639 18323 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 06:07:13.543547 18238 master_runner.cc:428] Detected that this master 0a9e7156fd5c4d399486dc5932b407b2 is joining an existing cluster
I20260620 06:07:13.543615 18238 master_runner.cc:432] Initiating AddMaster RPC to add 127.11.102.61:39977
I20260620 06:07:13.553866 17813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:07:13.545493 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54628 (local address 127.11.102.62:44777)
0620 06:07:13.545612 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0620 06:07:13.545616 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 06:07:13.545852 (+ 236us) server_negotiation.cc:407] Connection header received
0620 06:07:13.546313 (+ 461us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:07:13.546319 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:07:13.546371 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:07:13.546421 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:07:13.546971 (+ 550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:07:13.547546 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:07:13.548821 (+ 1275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:07:13.549159 (+ 338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:07:13.549235 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:07:13.550567 (+ 1332us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:07:13.550573 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:07:13.550576 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:07:13.550602 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:07:13.552175 (+ 1573us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:07:13.552704 (+ 529us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:07:13.552707 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:07:13.552711 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:07:13.552794 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:07:13.553069 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:07:13.553074 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:07:13.553076 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:07:13.553253 (+ 177us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:07:13.553362 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0620 06:07:13.553608 (+ 246us) server_negotiation.cc:299] Negotiation successful
0620 06:07:13.553715 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260620 06:07:13.553881 18329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:07:13.545469 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.102.62:44777 (local address 127.0.0.1:54628)
0620 06:07:13.545772 (+ 303us) negotiation.cc:107] Waiting for socket to connect
0620 06:07:13.545786 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 06:07:13.546200 (+ 414us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:07:13.546551 (+ 351us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:07:13.546557 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:07:13.546757 (+ 200us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:07:13.546860 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:07:13.546865 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:07:13.547684 (+ 819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:07:13.547688 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:07:13.548672 (+ 984us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:07:13.548681 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:07:13.549376 (+ 695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:07:13.549380 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:07:13.549471 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:07:13.549966 (+ 495us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:07:13.549984 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:07:13.550411 (+ 427us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:07:13.552311 (+ 1900us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:07:13.552314 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:07:13.552316 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:07:13.552607 (+ 291us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:07:13.552893 (+ 286us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:07:13.552895 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:07:13.552897 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:07:13.552981 (+ 84us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:07:13.553354 (+ 373us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:07:13.553357 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:07:13.553452 (+ 95us) client_negotiation.cc:769] Sending connection context
0620 06:07:13.553559 (+ 107us) client_negotiation.cc:240] Negotiation successful
0620 06:07:13.553692 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":209,"mutex_wait_us":35,"thread_start_us":92,"threads_started":1}
I20260620 06:07:13.554147 18309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:07:13.545859 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52458 (local address 127.11.102.61:39977)
0620 06:07:13.545951 (+ 92us) server_negotiation.cc:206] Beginning negotiation
0620 06:07:13.545957 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 06:07:13.545967 (+ 10us) server_negotiation.cc:407] Connection header received
0620 06:07:13.546000 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:07:13.546002 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:07:13.546030 (+ 28us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:07:13.546112 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:07:13.546643 (+ 531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:07:13.547451 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:07:13.548330 (+ 879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:07:13.548648 (+ 318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:07:13.548760 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:07:13.550602 (+ 1842us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:07:13.550605 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:07:13.550608 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:07:13.550630 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:07:13.552629 (+ 1999us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:07:13.553073 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:07:13.553077 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:07:13.553080 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:07:13.553143 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:07:13.553392 (+ 249us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:07:13.553394 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:07:13.553396 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:07:13.553568 (+ 172us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:07:13.553630 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0620 06:07:13.553906 (+ 276us) server_negotiation.cc:299] Negotiation successful
0620 06:07:13.554003 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":81,"server-negotiator.queue_time_us":27,"spinlock_wait_cycles":4480}
I20260620 06:07:13.554574 18330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:07:13.545547 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.102.61:39977 (local address 127.0.0.1:52458)
0620 06:07:13.545788 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0620 06:07:13.545801 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 06:07:13.545882 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:07:13.546135 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:07:13.546138 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:07:13.546383 (+ 245us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:07:13.546511 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:07:13.546519 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:07:13.547587 (+ 1068us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:07:13.547593 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:07:13.548217 (+ 624us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:07:13.548225 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:07:13.548770 (+ 545us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:07:13.548773 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:07:13.548875 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:07:13.550127 (+ 1252us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:07:13.550149 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:07:13.550472 (+ 323us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:07:13.552739 (+ 2267us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:07:13.552741 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:07:13.552743 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:07:13.552978 (+ 235us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:07:13.553231 (+ 253us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:07:13.553233 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:07:13.553235 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:07:13.553310 (+ 75us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:07:13.553684 (+ 374us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:07:13.553687 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:07:13.553752 (+ 65us) client_negotiation.cc:769] Sending connection context
0620 06:07:13.554309 (+ 557us) client_negotiation.cc:240] Negotiation successful
0620 06:07:13.554421 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":192,"mutex_wait_us":1,"thread_start_us":94,"threads_started":1}
Jun 20 06:07:13 dist-test-slave-9hm5 krb5kdc[17729](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935632, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.102.62@KRBTEST.COM for oryx/127.11.102.61@KRBTEST.COM
I20260620 06:07:13.567027 18309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:07:13.558475 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52460 (local address 127.11.102.61:39977)
0620 06:07:13.558580 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0620 06:07:13.558585 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 06:07:13.558888 (+ 303us) server_negotiation.cc:407] Connection header received
0620 06:07:13.559097 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:07:13.559100 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:07:13.559147 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:07:13.559235 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:07:13.559949 (+ 714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:07:13.560418 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:07:13.561291 (+ 873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:07:13.561515 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:07:13.561623 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:07:13.563976 (+ 2353us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:07:13.563980 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:07:13.563982 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:07:13.564011 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:07:13.565380 (+ 1369us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:07:13.565839 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:07:13.565844 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:07:13.565846 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:07:13.565921 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:07:13.566193 (+ 272us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:07:13.566198 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:07:13.566200 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:07:13.566359 (+ 159us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:07:13.566466 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0620 06:07:13.566781 (+ 315us) server_negotiation.cc:299] Negotiation successful
0620 06:07:13.566881 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260620 06:07:13.567085 18331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:07:13.558432 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.102.61:39977 (local address 127.0.0.1:52460)
0620 06:07:13.558761 (+ 329us) negotiation.cc:107] Waiting for socket to connect
0620 06:07:13.558802 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0620 06:07:13.558958 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:07:13.559272 (+ 314us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:07:13.559278 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:07:13.559686 (+ 408us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:07:13.559830 (+ 144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:07:13.559838 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:07:13.560522 (+ 684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:07:13.560527 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:07:13.561148 (+ 621us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:07:13.561156 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:07:13.561624 (+ 468us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:07:13.561627 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:07:13.561738 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:07:13.562272 (+ 534us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:07:13.562283 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:07:13.563841 (+ 1558us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:07:13.565493 (+ 1652us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:07:13.565503 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:07:13.565509 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:07:13.565732 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:07:13.566013 (+ 281us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:07:13.566017 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:07:13.566020 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:07:13.566105 (+ 85us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:07:13.566465 (+ 360us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:07:13.566473 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:07:13.566591 (+ 118us) client_negotiation.cc:769] Sending connection context
0620 06:07:13.566731 (+ 140us) client_negotiation.cc:240] Negotiation successful
0620 06:07:13.566922 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":91,"threads_started":1}
I20260620 06:07:13.568403 17774 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.11.102.61:39977: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "0a9e7156fd5c4d399486dc5932b407b2"
member_type: NON_VOTER
last_known_addr {
host: "127.11.102.61"
port: 39977
}
attrs {
promote: true
}
}
dest_uuid: "d7d4500dc38a48a1858293d72d1fcfc2"
cas_config_opid_index: -1
I20260620 06:07:13.568984 17774 consensus_queue.cc:237] T 00000000000000000000000000000000 P d7d4500dc38a48a1858293d72d1fcfc2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "d7d4500dc38a48a1858293d72d1fcfc2" member_type: VOTER last_known_addr { host: "127.11.102.62" port: 44777 } } peers { permanent_uuid: "0a9e7156fd5c4d399486dc5932b407b2" member_type: NON_VOTER last_known_addr { host: "127.11.102.61" port: 39977 } attrs { promote: true } }
I20260620 06:07:13.570776 18332 raft_consensus.cc:2932] T 00000000000000000000000000000000 P d7d4500dc38a48a1858293d72d1fcfc2 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 0a9e7156fd5c4d399486dc5932b407b2 (127.11.102.61) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "d7d4500dc38a48a1858293d72d1fcfc2" member_type: VOTER last_known_addr { host: "127.11.102.62" port: 44777 } } peers { permanent_uuid: "0a9e7156fd5c4d399486dc5932b407b2" member_type: NON_VOTER last_known_addr { host: "127.11.102.61" port: 39977 } attrs { promote: true } } }
I20260620 06:07:13.571149 18280 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0a9e7156fd5c4d399486dc5932b407b2 [term 0 FOLLOWER]: Advancing to term 1
I20260620 06:07:13.572489 18280 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 0a9e7156fd5c4d399486dc5932b407b2 [term 1 FOLLOWER]: Refusing update from remote peer d7d4500dc38a48a1858293d72d1fcfc2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260620 06:07:13.572525 18332 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.11.102.61:39977
I20260620 06:07:13.573161 18238 master.cc:537] Master@127.11.102.61:39977 shutting down...
I20260620 06:07:13.573179 18335 consensus_queue.cc:1035] T 00000000000000000000000000000000 P d7d4500dc38a48a1858293d72d1fcfc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a9e7156fd5c4d399486dc5932b407b2" member_type: NON_VOTER last_known_addr { host: "127.11.102.61" port: 39977 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 06:07:13.573599 18333 sys_catalog.cc:455] T 00000000000000000000000000000000 P d7d4500dc38a48a1858293d72d1fcfc2 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "d7d4500dc38a48a1858293d72d1fcfc2" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "d7d4500dc38a48a1858293d72d1fcfc2" member_type: VOTER last_known_addr { host: "127.11.102.62" port: 44777 } } peers { permanent_uuid: "0a9e7156fd5c4d399486dc5932b407b2" member_type: NON_VOTER last_known_addr { host: "127.11.102.61" port: 39977 } attrs { promote: true } } }
I20260620 06:07:13.573695 18333 sys_catalog.cc:458] T 00000000000000000000000000000000 P d7d4500dc38a48a1858293d72d1fcfc2 [sys.catalog]: This master's current role is: LEADER
W20260620 06:07:13.576362 17760 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
I20260620 06:07:13.577904 18238 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 0a9e7156fd5c4d399486dc5932b407b2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 06:07:13.578184 18238 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 0a9e7156fd5c4d399486dc5932b407b2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 06:07:13.578307 18238 tablet_replica.cc:330] T 00000000000000000000000000000000 P 0a9e7156fd5c4d399486dc5932b407b2: stopping tablet replica
WARNING: no policy specified for oryx/127.11.102.60@KRBTEST.COM; defaulting to no policy
I20260620 06:07:13.582007 18331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:07:13.581705 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.102.61:39977 (local address 127.0.0.1:52464)
0620 06:07:13.581872 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0620 06:07:13.581919 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.102.61:39977: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109}
W20260620 06:07:13.582176 17760 consensus_peers.cc:487] T 00000000000000000000000000000000 P d7d4500dc38a48a1858293d72d1fcfc2 -> Peer 0a9e7156fd5c4d399486dc5932b407b2 (127.11.102.61:39977): Couldn't send request to peer 0a9e7156fd5c4d399486dc5932b407b2. Status: Network error: Client connection negotiation failed: client connection to 127.11.102.61:39977: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.11.102.60@KRBTEST.COM" created.
I20260620 06:07:13.592890 18238 master.cc:552] Master@127.11.102.61:39977 shutdown complete.
I20260620 06:07:13.592994 18238 master_runner.cc:305] Clearing existing system tablet
I20260620 06:07:13.593327 18238 fs_manager.cc:359] Metadata directory not provided
I20260620 06:07:13.593433 18238 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:07:13.594786 18238 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 06:07:13.595717 18341 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:07:13.595993 18238 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935603120653-11672-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:07:13.596136 18238 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 06:07:13.596171 18238 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935603120653-11672-0/minicluster-data/master-1/data,/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935603120653-11672-0/minicluster-data/master-1/wal
uuid: "0a9e7156fd5c4d399486dc5932b407b2"
format_stamp: "Formatted at 2026-06-20 06:07:13 on dist-test-slave-9hm5"
server_key: "02d4284fc9e50f19db1fb4050a8eebb4"
server_key_iv: "182e2cca2183c69f2163b6f3e9330b58"
server_key_version: "encryptionkey@0"
I20260620 06:07:13.597982 18238 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 0a9e7156fd5c4d399486dc5932b407b2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 06:07:13.601245 18238 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 0a9e7156fd5c4d399486dc5932b407b2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260620 06:07:13.601352 18238 log.cc:1196] T 00000000000000000000000000000000 P 0a9e7156fd5c4d399486dc5932b407b2: Deleting WAL directory at /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935603120653-11672-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260620 06:07:13.601645 18238 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 0a9e7156fd5c4d399486dc5932b407b2: Deleting consensus metadata
I20260620 06:07:13.601835 18238 master_runner.cc:315] Copying system tablet from 127.11.102.62:44777
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.11.102.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935603120653-11672-0/krb5kdc/oryx_127.11.102.60.keytab.
Entry for principal oryx/127.11.102.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935603120653-11672-0/krb5kdc/oryx_127.11.102.60.keytab.
I20260620 06:07:13.602869 11672 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9HMp43/build/debug/bin/kudu
/tmp/dist-test-task9HMp43/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935603120653-11672-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935603120653-11672-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935603120653-11672-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935603120653-11672-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.102.60:41691
--webserver_interface=127.11.102.60
--webserver_port=0
--keytab_file=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935603120653-11672-0/krb5kdc/oryx_127.11.102.60.keytab
--principal=oryx/127.11.102.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.102.20:34945
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.102.62:44777,127.11.102.61:39977,127.11.102.60:41691
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935603120653-11672-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935603120653-11672-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935603120653-11672-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 06:07:13.604519 18238 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 0a9e7156fd5c4d399486dc5932b407b2: tablet copy: Beginning tablet copy session from remote peer at address 127.11.102.62:44777
I20260620 06:07:13.612634 17813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:07:13.605108 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54642 (local address 127.11.102.62:44777)
0620 06:07:13.605228 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0620 06:07:13.605233 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 06:07:13.605477 (+ 244us) server_negotiation.cc:407] Connection header received
0620 06:07:13.605629 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:07:13.605632 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:07:13.605678 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:07:13.605779 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:07:13.606408 (+ 629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:07:13.607013 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:07:13.607936 (+ 923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:07:13.608218 (+ 282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:07:13.608332 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:07:13.609464 (+ 1132us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:07:13.609468 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:07:13.609471 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:07:13.609497 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:07:13.611023 (+ 1526us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:07:13.611458 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:07:13.611462 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:07:13.611464 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:07:13.611547 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:07:13.611834 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:07:13.611839 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:07:13.611840 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:07:13.612015 (+ 175us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:07:13.612118 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0620 06:07:13.612372 (+ 254us) server_negotiation.cc:299] Negotiation successful
0620 06:07:13.612477 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260620 06:07:13.612643 18348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:07:13.605061 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.102.62:44777 (local address 127.0.0.1:54642)
0620 06:07:13.605384 (+ 323us) negotiation.cc:107] Waiting for socket to connect
0620 06:07:13.605396 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 06:07:13.605524 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:07:13.605793 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:07:13.605796 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:07:13.605995 (+ 199us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:07:13.606288 (+ 293us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:07:13.606295 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:07:13.607123 (+ 828us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:07:13.607128 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:07:13.607820 (+ 692us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:07:13.607829 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:07:13.608333 (+ 504us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:07:13.608337 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:07:13.608480 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:07:13.609002 (+ 522us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:07:13.609014 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:07:13.609328 (+ 314us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:07:13.611150 (+ 1822us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:07:13.611153 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:07:13.611156 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:07:13.611360 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:07:13.611652 (+ 292us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:07:13.611656 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:07:13.611658 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:07:13.611745 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:07:13.612107 (+ 362us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:07:13.612110 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:07:13.612210 (+ 100us) client_negotiation.cc:769] Sending connection context
0620 06:07:13.612321 (+ 111us) client_negotiation.cc:240] Negotiation successful
0620 06:07:13.612459 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":255,"thread_start_us":89,"threads_started":1}
I20260620 06:07:13.613492 17794 tablet_copy_service.cc:140] P d7d4500dc38a48a1858293d72d1fcfc2: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 0a9e7156fd5c4d399486dc5932b407b2 ({username='oryx', principal='oryx/127.11.102.61@KRBTEST.COM'} at 127.0.0.1:54642)
I20260620 06:07:13.613662 17794 tablet_copy_service.cc:161] P d7d4500dc38a48a1858293d72d1fcfc2: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 0a9e7156fd5c4d399486dc5932b407b2 at {username='oryx', principal='oryx/127.11.102.61@KRBTEST.COM'} at 127.0.0.1:54642: session id = 0a9e7156fd5c4d399486dc5932b407b2-00000000000000000000000000000000
I20260620 06:07:13.615065 17794 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P d7d4500dc38a48a1858293d72d1fcfc2: Tablet Copy: opened 0 blocks and 1 log segments
I20260620 06:07:13.616501 18238 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260620 06:07:13.618316 18238 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 0a9e7156fd5c4d399486dc5932b407b2: tablet copy: Starting download of 0 data blocks...
I20260620 06:07:13.618510 18238 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 0a9e7156fd5c4d399486dc5932b407b2: tablet copy: Starting download of 1 WAL segments...
I20260620 06:07:13.620062 18238 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 0a9e7156fd5c4d399486dc5932b407b2: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260620 06:07:13.621462 17794 tablet_copy_service.cc:342] P d7d4500dc38a48a1858293d72d1fcfc2: Request end of tablet copy session 0a9e7156fd5c4d399486dc5932b407b2-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.11.102.61@KRBTEST.COM'} at 127.0.0.1:54642
I20260620 06:07:13.621579 17794 tablet_copy_service.cc:433] P d7d4500dc38a48a1858293d72d1fcfc2: ending tablet copy session 0a9e7156fd5c4d399486dc5932b407b2-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 0a9e7156fd5c4d399486dc5932b407b2
I20260620 06:07:13.628008 18238 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 06:07:13.629611 18349 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:07:13.629737 18350 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:07:13.629976 18352 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:07:13.630264 18238 server_base.cc:1034] running on GCE node
I20260620 06:07:13.630442 18238 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:07:13.630766 18238 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:07:13.631896 18238 hybrid_clock.cc:648] HybridClock initialized: now 1781935633631888 us; error 27 us; skew 500 ppm
Jun 20 06:07:13 dist-test-slave-9hm5 krb5kdc[17729](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935633, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.102.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 06:07:13.633291 18238 init.cc:377] Logged in from keytab as oryx/127.11.102.61@KRBTEST.COM (short username oryx)
I20260620 06:07:13.633735 18238 webserver.cc:460] Webserver started at http://127.11.102.61:41195/ using document root <none> and password file <none>
I20260620 06:07:13.633912 18238 fs_manager.cc:359] Metadata directory not provided
I20260620 06:07:13.633965 18238 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:07:13.634992 18238 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 06:07:13.635912 18359 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:07:13.636128 18238 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 06:07:13.636215 18238 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935603120653-11672-0/minicluster-data/master-1/data,/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935603120653-11672-0/minicluster-data/master-1/wal
uuid: "0a9e7156fd5c4d399486dc5932b407b2"
format_stamp: "Formatted at 2026-06-20 06:07:13 on dist-test-slave-9hm5"
server_key: "02d4284fc9e50f19db1fb4050a8eebb4"
server_key_iv: "182e2cca2183c69f2163b6f3e9330b58"
server_key_version: "encryptionkey@0"
I20260620 06:07:13.636317 18238 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935603120653-11672-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935603120653-11672-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935603120653-11672-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:07:13.643683 18238 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:07:13.645942 18238 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:07:13.652884 18238 rpc_server.cc:307] RPC server started. Bound to: 127.11.102.61:39977
I20260620 06:07:13.653426 18238 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935603120653-11672-0/minicluster-data/master-1/data/info.pb
I20260620 06:07:13.654470 18411 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.102.61:39977 every 8 connection(s)
I20260620 06:07:13.655169 18412 sys_catalog.cc:263] Verifying existing consensus state
I20260620 06:07:13.655982 18412 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0a9e7156fd5c4d399486dc5932b407b2: Bootstrap starting.
I20260620 06:07:13.664013 18412 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0a9e7156fd5c4d399486dc5932b407b2: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 06:07:13.664299 18412 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0a9e7156fd5c4d399486dc5932b407b2: Bootstrap complete.
I20260620 06:07:13.664749 18412 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0a9e7156fd5c4d399486dc5932b407b2 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "d7d4500dc38a48a1858293d72d1fcfc2" member_type: VOTER last_known_addr { host: "127.11.102.62" port: 44777 } } peers { permanent_uuid: "0a9e7156fd5c4d399486dc5932b407b2" member_type: NON_VOTER last_known_addr { host: "127.11.102.61" port: 39977 } attrs { promote: true } }
I20260620 06:07:13.664877 18412 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0a9e7156fd5c4d399486dc5932b407b2 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 0a9e7156fd5c4d399486dc5932b407b2, State: Initialized, Role: LEARNER
I20260620 06:07:13.664989 18412 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0a9e7156fd5c4d399486dc5932b407b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "d7d4500dc38a48a1858293d72d1fcfc2" member_type: VOTER last_known_addr { host: "127.11.102.62" port: 44777 } } peers { permanent_uuid: "0a9e7156fd5c4d399486dc5932b407b2" member_type: NON_VOTER last_known_addr { host: "127.11.102.61" port: 39977 } attrs { promote: true } }
I20260620 06:07:13.665311 18415 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0a9e7156fd5c4d399486dc5932b407b2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "d7d4500dc38a48a1858293d72d1fcfc2" member_type: VOTER last_known_addr { host: "127.11.102.62" port: 44777 } } peers { permanent_uuid: "0a9e7156fd5c4d399486dc5932b407b2" member_type: NON_VOTER last_known_addr { host: "127.11.102.61" port: 39977 } attrs { promote: true } } }
I20260620 06:07:13.665463 18415 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0a9e7156fd5c4d399486dc5932b407b2 [sys.catalog]: This master's current role is: LEARNER
I20260620 06:07:13.665485 18412 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0a9e7156fd5c4d399486dc5932b407b2 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 06:07:13.668803 18426 catalog_manager.cc:1259] Loaded cluster ID: bcc977c4637644dab45ae102d36c03ef
I20260620 06:07:13.668876 18426 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 0a9e7156fd5c4d399486dc5932b407b2: loading cluster ID for follower catalog manager: success
I20260620 06:07:13.670399 18426 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 0a9e7156fd5c4d399486dc5932b407b2: acquiring CA information for follower catalog manager: success
I20260620 06:07:13.671420 18426 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 0a9e7156fd5c4d399486dc5932b407b2: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260620 06:07:13.747474 18346 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:07:13.747828 18346 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:07:13.747908 18346 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:07:13.751969 18346 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 06:07:13.752035 18346 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:07:13.752074 18346 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:07:13.752127 18346 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 06:07:13.752164 18346 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 06:07:13.757246 18346 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.102.20:34945
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935603120653-11672-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935603120653-11672-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.11.102.62:44777,127.11.102.61:39977,127.11.102.60:41691
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935603120653-11672-0/krb5kdc/oryx_127.11.102.60.keytab
--principal=oryx/127.11.102.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.102.60:41691
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935603120653-11672-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.11.102.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18346
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935603120653-11672-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 05:28:48 UTC on ccycloud.quasar-fixhms.root.comops.site
build id 4457020
I20260620 06:07:13.758216 18346 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:07:13.759161 18346 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 06:07:13.767825 18431 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:07:13.767825 18434 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:07:13.767892 18432 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:07:13.768107 18346 server_base.cc:1034] running on GCE node
I20260620 06:07:13.768730 18346 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:07:13.769729 18346 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:07:13.770906 18346 hybrid_clock.cc:648] HybridClock initialized: now 1781935633770886 us; error 31 us; skew 500 ppm
Jun 20 06:07:13 dist-test-slave-9hm5 krb5kdc[17729](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935633, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.102.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 06:07:13.774204 18346 init.cc:377] Logged in from keytab as oryx/127.11.102.60@KRBTEST.COM (short username oryx)
I20260620 06:07:13.775240 18346 webserver.cc:460] Webserver started at http://127.11.102.60:43607/ using document root <none> and password file <none>
I20260620 06:07:13.775871 18346 fs_manager.cc:359] Metadata directory not provided
I20260620 06:07:13.775934 18346 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:07:13.776166 18346 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 06:07:13.778162 18346 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935603120653-11672-0/minicluster-data/master-2/data/instance:
uuid: "ec929d712e20476684cecae76f307d8e"
format_stamp: "Formatted at 2026-06-20 06:07:13 on dist-test-slave-9hm5"
server_key: "8ccea1aa0515b1fbeea42eeee4193cc7"
server_key_iv: "7dd9d3a44a97356d9d87a40c9a72f8e2"
server_key_version: "encryptionkey@0"
I20260620 06:07:13.778728 18346 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935603120653-11672-0/minicluster-data/master-2/wal/instance:
uuid: "ec929d712e20476684cecae76f307d8e"
format_stamp: "Formatted at 2026-06-20 06:07:13 on dist-test-slave-9hm5"
server_key: "8ccea1aa0515b1fbeea42eeee4193cc7"
server_key_iv: "7dd9d3a44a97356d9d87a40c9a72f8e2"
server_key_version: "encryptionkey@0"
I20260620 06:07:13.782404 18346 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260620 06:07:13.784834 18441 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:07:13.786149 18346 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260620 06:07:13.786262 18346 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935603120653-11672-0/minicluster-data/master-2/data,/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935603120653-11672-0/minicluster-data/master-2/wal
uuid: "ec929d712e20476684cecae76f307d8e"
format_stamp: "Formatted at 2026-06-20 06:07:13 on dist-test-slave-9hm5"
server_key: "8ccea1aa0515b1fbeea42eeee4193cc7"
server_key_iv: "7dd9d3a44a97356d9d87a40c9a72f8e2"
server_key_version: "encryptionkey@0"
I20260620 06:07:13.786365 18346 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935603120653-11672-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935603120653-11672-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935603120653-11672-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:07:13.799574 18346 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:07:13.802858 18346 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:07:13.803021 18346 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:07:13.810642 18346 rpc_server.cc:307] RPC server started. Bound to: 127.11.102.60:41691
I20260620 06:07:13.810932 18493 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.102.60:41691 every 8 connection(s)
I20260620 06:07:13.811589 18346 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935603120653-11672-0/minicluster-data/master-2/data/info.pb
I20260620 06:07:13.811882 11672 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9HMp43/build/debug/bin/kudu as pid 18346
I20260620 06:07:13.814424 18494 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 06:07:13.818064 18494 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.11.102.62" port: 44777 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 06:07:13 dist-test-slave-9hm5 krb5kdc[17729](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935632, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.11.102.60@KRBTEST.COM
Jun 20 06:07:13 dist-test-slave-9hm5 krb5kdc[17729](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935633, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.102.60@KRBTEST.COM for oryx/127.11.102.62@KRBTEST.COM
I20260620 06:07:13.826517 18496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:07:13.813383 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56080 (local address 127.11.102.60:41691)
0620 06:07:13.813753 (+ 370us) server_negotiation.cc:206] Beginning negotiation
0620 06:07:13.813770 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0620 06:07:13.813810 (+ 40us) server_negotiation.cc:407] Connection header received
0620 06:07:13.814436 (+ 626us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:07:13.814451 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:07:13.814883 (+ 432us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:07:13.815589 (+ 706us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:07:13.816381 (+ 792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:07:13.817340 (+ 959us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:07:13.818174 (+ 834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:07:13.818683 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:07:13.818840 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:07:13.821868 (+ 3028us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:07:13.821872 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:07:13.821881 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:07:13.821904 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:07:13.823751 (+ 1847us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:07:13.824210 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:07:13.824216 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:07:13.824221 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:07:13.824350 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:07:13.824620 (+ 270us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:07:13.824625 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:07:13.824627 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:07:13.824993 (+ 366us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:07:13.825197 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0620 06:07:13.825481 (+ 284us) server_negotiation.cc:299] Negotiation successful
0620 06:07:13.825778 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":247,"thread_start_us":133,"threads_started":1}
I20260620 06:07:13.829493 17813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:07:13.820744 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54644 (local address 127.11.102.62:44777)
0620 06:07:13.820868 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0620 06:07:13.820871 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 06:07:13.820884 (+ 13us) server_negotiation.cc:407] Connection header received
0620 06:07:13.820918 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:07:13.820920 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:07:13.820955 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:07:13.821020 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:07:13.821783 (+ 763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:07:13.822465 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:07:13.823777 (+ 1312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:07:13.824130 (+ 353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:07:13.824234 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:07:13.826415 (+ 2181us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:07:13.826419 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:07:13.826421 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:07:13.826447 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:07:13.827943 (+ 1496us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:07:13.828371 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:07:13.828374 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:07:13.828376 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:07:13.828463 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:07:13.828798 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:07:13.828800 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:07:13.828801 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:07:13.828961 (+ 160us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:07:13.829035 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0620 06:07:13.829303 (+ 268us) server_negotiation.cc:299] Negotiation successful
0620 06:07:13.829390 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260620 06:07:13.829519 18497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:07:13.819805 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.102.62:44777 (local address 127.0.0.1:54644)
0620 06:07:13.820590 (+ 785us) negotiation.cc:107] Waiting for socket to connect
0620 06:07:13.820627 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0620 06:07:13.820763 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:07:13.821103 (+ 340us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:07:13.821109 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:07:13.821518 (+ 409us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:07:13.821640 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:07:13.821645 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:07:13.822581 (+ 936us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:07:13.822586 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:07:13.823663 (+ 1077us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:07:13.823674 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:07:13.824231 (+ 557us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:07:13.824234 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:07:13.824345 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:07:13.824901 (+ 556us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:07:13.824921 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:07:13.826282 (+ 1361us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:07:13.828047 (+ 1765us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:07:13.828054 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:07:13.828059 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:07:13.828277 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:07:13.828556 (+ 279us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:07:13.828559 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:07:13.828561 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:07:13.828653 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:07:13.829047 (+ 394us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:07:13.829052 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:07:13.829139 (+ 87us) client_negotiation.cc:769] Sending connection context
0620 06:07:13.829250 (+ 111us) client_negotiation.cc:240] Negotiation successful
0620 06:07:13.829372 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":637,"thread_start_us":129,"threads_started":1}
I20260620 06:07:13.830415 18494 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.11.102.61" port: 39977 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 06:07:13 dist-test-slave-9hm5 krb5kdc[17729](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935633, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.102.60@KRBTEST.COM for oryx/127.11.102.61@KRBTEST.COM
I20260620 06:07:13.838701 18497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:07:13.830861 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.102.61:39977 (local address 127.0.0.1:52472)
0620 06:07:13.830962 (+ 101us) negotiation.cc:107] Waiting for socket to connect
0620 06:07:13.830977 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 06:07:13.831045 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:07:13.831449 (+ 404us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:07:13.831452 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:07:13.831643 (+ 191us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:07:13.831744 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:07:13.831749 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:07:13.832657 (+ 908us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:07:13.832671 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:07:13.833225 (+ 554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:07:13.833234 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:07:13.833823 (+ 589us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:07:13.833826 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:07:13.833895 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:07:13.834264 (+ 369us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:07:13.834275 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:07:13.835371 (+ 1096us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:07:13.837253 (+ 1882us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:07:13.837255 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:07:13.837257 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:07:13.837438 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:07:13.837766 (+ 328us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:07:13.837768 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:07:13.837770 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:07:13.837840 (+ 70us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:07:13.838174 (+ 334us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:07:13.838177 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:07:13.838276 (+ 99us) client_negotiation.cc:769] Sending connection context
0620 06:07:13.838379 (+ 103us) client_negotiation.cc:240] Negotiation successful
0620 06:07:13.838521 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":43}
I20260620 06:07:13.838737 18498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:07:13.830989 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52472 (local address 127.11.102.61:39977)
0620 06:07:13.831254 (+ 265us) server_negotiation.cc:206] Beginning negotiation
0620 06:07:13.831258 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 06:07:13.831268 (+ 10us) server_negotiation.cc:407] Connection header received
0620 06:07:13.831301 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:07:13.831303 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:07:13.831340 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:07:13.831440 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:07:13.831864 (+ 424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:07:13.832546 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:07:13.833347 (+ 801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:07:13.833714 (+ 367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:07:13.833830 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:07:13.835492 (+ 1662us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:07:13.835494 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:07:13.835496 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:07:13.835519 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:07:13.837141 (+ 1622us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:07:13.837594 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:07:13.837600 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:07:13.837606 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:07:13.837685 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:07:13.837916 (+ 231us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:07:13.837918 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:07:13.837919 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:07:13.838074 (+ 155us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:07:13.838147 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0620 06:07:13.838473 (+ 326us) server_negotiation.cc:299] Negotiation successful
0620 06:07:13.838592 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":199,"thread_start_us":98,"threads_started":1}
I20260620 06:07:13.839514 18494 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.11.102.60" port: 41691 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 06:07:13 dist-test-slave-9hm5 krb5kdc[17729](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935633, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.102.60@KRBTEST.COM for oryx/127.11.102.60@KRBTEST.COM
I20260620 06:07:13.846728 18497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:07:13.839967 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.102.60:41691 (local address 127.0.0.1:56086)
0620 06:07:13.840067 (+ 100us) negotiation.cc:107] Waiting for socket to connect
0620 06:07:13.840082 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 06:07:13.840151 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:07:13.840330 (+ 179us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:07:13.840332 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:07:13.840444 (+ 112us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:07:13.840506 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:07:13.840509 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:07:13.841504 (+ 995us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:07:13.841507 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:07:13.841987 (+ 480us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:07:13.841994 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:07:13.842377 (+ 383us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:07:13.842379 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:07:13.842431 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:07:13.842937 (+ 506us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:07:13.842948 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:07:13.844062 (+ 1114us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:07:13.845606 (+ 1544us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:07:13.845608 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:07:13.845610 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:07:13.845777 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:07:13.845993 (+ 216us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:07:13.845995 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:07:13.845996 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:07:13.846059 (+ 63us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:07:13.846343 (+ 284us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:07:13.846345 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:07:13.846408 (+ 63us) client_negotiation.cc:769] Sending connection context
0620 06:07:13.846480 (+ 72us) client_negotiation.cc:240] Negotiation successful
0620 06:07:13.846563 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":43}
I20260620 06:07:13.846743 18496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:07:13.840052 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56086 (local address 127.11.102.60:41691)
0620 06:07:13.840152 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0620 06:07:13.840156 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 06:07:13.840194 (+ 38us) server_negotiation.cc:407] Connection header received
0620 06:07:13.840221 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:07:13.840223 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:07:13.840252 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:07:13.840305 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:07:13.840589 (+ 284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:07:13.841053 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:07:13.842081 (+ 1028us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:07:13.842289 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:07:13.842362 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:07:13.844172 (+ 1810us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:07:13.844175 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:07:13.844177 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:07:13.844196 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:07:13.845524 (+ 1328us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:07:13.845852 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:07:13.845855 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:07:13.845856 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:07:13.845920 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:07:13.846127 (+ 207us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:07:13.846130 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:07:13.846131 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:07:13.846272 (+ 141us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:07:13.846340 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0620 06:07:13.846531 (+ 191us) server_negotiation.cc:299] Negotiation successful
0620 06:07:13.846609 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260620 06:07:13.849510 18494 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ec929d712e20476684cecae76f307d8e: Bootstrap starting.
I20260620 06:07:13.851742 18494 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ec929d712e20476684cecae76f307d8e: Neither blocks nor log segments found. Creating new log.
I20260620 06:07:13.852434 18494 log.cc:824] T 00000000000000000000000000000000 P ec929d712e20476684cecae76f307d8e: Log is configured to *not* fsync() on all Append() calls
I20260620 06:07:13.853606 17965 heartbeater.cc:503] Master 127.11.102.62:44777 was elected leader, sending a full tablet report...
I20260620 06:07:13.854290 18494 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ec929d712e20476684cecae76f307d8e: No bootstrap required, opened a new log
I20260620 06:07:13.857218 18494 raft_consensus.cc:348] T 00000000000000000000000000000000 P ec929d712e20476684cecae76f307d8e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7d4500dc38a48a1858293d72d1fcfc2" member_type: VOTER last_known_addr { host: "127.11.102.62" port: 44777 } } peers { permanent_uuid: "0a9e7156fd5c4d399486dc5932b407b2" member_type: VOTER last_known_addr { host: "127.11.102.61" port: 39977 } } peers { permanent_uuid: "ec929d712e20476684cecae76f307d8e" member_type: VOTER last_known_addr { host: "127.11.102.60" port: 41691 } }
I20260620 06:07:13.857429 18494 raft_consensus.cc:374] T 00000000000000000000000000000000 P ec929d712e20476684cecae76f307d8e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 06:07:13.857512 18494 raft_consensus.cc:729] T 00000000000000000000000000000000 P ec929d712e20476684cecae76f307d8e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ec929d712e20476684cecae76f307d8e, State: Initialized, Role: FOLLOWER
I20260620 06:07:13.857924 18494 consensus_queue.cc:260] T 00000000000000000000000000000000 P ec929d712e20476684cecae76f307d8e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7d4500dc38a48a1858293d72d1fcfc2" member_type: VOTER last_known_addr { host: "127.11.102.62" port: 44777 } } peers { permanent_uuid: "0a9e7156fd5c4d399486dc5932b407b2" member_type: VOTER last_known_addr { host: "127.11.102.61" port: 39977 } } peers { permanent_uuid: "ec929d712e20476684cecae76f307d8e" member_type: VOTER last_known_addr { host: "127.11.102.60" port: 41691 } }
I20260620 06:07:13.858346 18500 sys_catalog.cc:455] T 00000000000000000000000000000000 P ec929d712e20476684cecae76f307d8e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7d4500dc38a48a1858293d72d1fcfc2" member_type: VOTER last_known_addr { host: "127.11.102.62" port: 44777 } } peers { permanent_uuid: "0a9e7156fd5c4d399486dc5932b407b2" member_type: VOTER last_known_addr { host: "127.11.102.61" port: 39977 } } peers { permanent_uuid: "ec929d712e20476684cecae76f307d8e" member_type: VOTER last_known_addr { host: "127.11.102.60" port: 41691 } } }
I20260620 06:07:13.858461 18500 sys_catalog.cc:458] T 00000000000000000000000000000000 P ec929d712e20476684cecae76f307d8e [sys.catalog]: This master's current role is: FOLLOWER
I20260620 06:07:13.859026 18494 sys_catalog.cc:564] T 00000000000000000000000000000000 P ec929d712e20476684cecae76f307d8e [sys.catalog]: configured and running, proceeding with master startup.
W20260620 06:07:13.863572 18511 catalog_manager.cc:1558] T 00000000000000000000000000000000 P ec929d712e20476684cecae76f307d8e: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 06:07:13.863682 18511 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 06:07:13.866752 18346 master_runner.cc:428] Detected that this master ec929d712e20476684cecae76f307d8e is joining an existing cluster
I20260620 06:07:13.866827 18346 master_runner.cc:432] Initiating AddMaster RPC to add 127.11.102.60:41691
I20260620 06:07:13.877898 17813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:07:13.869282 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54658 (local address 127.11.102.62:44777)
0620 06:07:13.869396 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0620 06:07:13.869402 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 06:07:13.869706 (+ 304us) server_negotiation.cc:407] Connection header received
0620 06:07:13.869888 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:07:13.869895 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:07:13.869960 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:07:13.870010 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:07:13.870730 (+ 720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:07:13.871526 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:07:13.872890 (+ 1364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:07:13.873208 (+ 318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:07:13.873275 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:07:13.874699 (+ 1424us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:07:13.874703 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:07:13.874705 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:07:13.874728 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:07:13.876301 (+ 1573us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:07:13.876749 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:07:13.876754 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:07:13.876756 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:07:13.876835 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:07:13.877105 (+ 270us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:07:13.877109 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:07:13.877111 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:07:13.877283 (+ 172us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:07:13.877387 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0620 06:07:13.877644 (+ 257us) server_negotiation.cc:299] Negotiation successful
0620 06:07:13.877751 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260620 06:07:13.877916 18517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:07:13.869236 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.102.62:44777 (local address 127.0.0.1:54658)
0620 06:07:13.869605 (+ 369us) negotiation.cc:107] Waiting for socket to connect
0620 06:07:13.869617 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 06:07:13.869773 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:07:13.870157 (+ 384us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:07:13.870163 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:07:13.870439 (+ 276us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:07:13.870585 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:07:13.870596 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:07:13.871677 (+ 1081us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:07:13.871680 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:07:13.872754 (+ 1074us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:07:13.872763 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:07:13.873445 (+ 682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:07:13.873451 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:07:13.873555 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:07:13.874202 (+ 647us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:07:13.874216 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:07:13.874556 (+ 340us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:07:13.876424 (+ 1868us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:07:13.876428 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:07:13.876430 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:07:13.876650 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:07:13.876923 (+ 273us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:07:13.876927 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:07:13.876929 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:07:13.877016 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:07:13.877375 (+ 359us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:07:13.877378 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:07:13.877485 (+ 107us) client_negotiation.cc:769] Sending connection context
0620 06:07:13.877592 (+ 107us) client_negotiation.cc:240] Negotiation successful
0620 06:07:13.877731 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":294,"mutex_wait_us":114,"spinlock_wait_cycles":2944,"thread_start_us":88,"threads_started":1}
I20260620 06:07:13.879802 11672 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9HMp43/build/debug/bin/kudu with pid 17836
Jun 20 06:07:13 dist-test-slave-9hm5 krb5kdc[17729](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935632, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.102.62@KRBTEST.COM for oryx/127.11.102.60@KRBTEST.COM
I20260620 06:07:13.888653 11672 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9HMp43/build/debug/bin/kudu
/tmp/dist-test-task9HMp43/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935603120653-11672-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935603120653-11672-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935603120653-11672-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935603120653-11672-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.102.1:37681
--local_ip_for_outbound_sockets=127.11.102.1
--tserver_master_addrs=127.11.102.62:44777
--webserver_port=36757
--webserver_interface=127.11.102.1
--keytab_file=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935603120653-11672-0/krb5kdc/oryx_127.11.102.1.keytab
--principal=oryx/127.11.102.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.102.20:34945
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.11.102.62:44777,127.11.102.61:39977
--tserver_master_addrs=127.11.102.62:44777,127.11.102.61:39977,127.11.102.60:41691 with env {KRB5CCNAME=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935603120653-11672-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935603120653-11672-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935603120653-11672-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 06:07:13.903908 18519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:07:13.870030 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.102.60:41691 (local address 127.0.0.1:56088)
0620 06:07:13.870541 (+ 511us) negotiation.cc:107] Waiting for socket to connect
0620 06:07:13.870555 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 06:07:13.870664 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:07:13.870927 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:07:13.870930 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:07:13.871094 (+ 164us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:07:13.871195 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:07:13.871201 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:07:13.872095 (+ 894us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:07:13.872098 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:07:13.872924 (+ 826us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:07:13.872931 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:07:13.873494 (+ 563us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:07:13.873496 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:07:13.873591 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:07:13.900208 (+ 26617us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:07:13.900224 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:07:13.900681 (+ 457us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:07:13.902445 (+ 1764us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:07:13.902447 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:07:13.902449 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:07:13.902673 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:07:13.902928 (+ 255us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:07:13.902931 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:07:13.902933 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:07:13.902999 (+ 66us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:07:13.903386 (+ 387us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:07:13.903388 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:07:13.903459 (+ 71us) client_negotiation.cc:769] Sending connection context
0620 06:07:13.903543 (+ 84us) client_negotiation.cc:240] Negotiation successful
0620 06:07:13.903705 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":450,"mutex_wait_us":33,"spinlock_wait_cycles":1280,"thread_start_us":76,"threads_started":1}
I20260620 06:07:13.904676 18496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:07:13.869871 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56088 (local address 127.11.102.60:41691)
0620 06:07:13.869963 (+ 92us) server_negotiation.cc:206] Beginning negotiation
0620 06:07:13.869967 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 06:07:13.870614 (+ 647us) server_negotiation.cc:407] Connection header received
0620 06:07:13.870749 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:07:13.870751 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:07:13.870784 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:07:13.870823 (+ 39us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:07:13.871297 (+ 474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:07:13.871988 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:07:13.873016 (+ 1028us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:07:13.873315 (+ 299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:07:13.873381 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:07:13.900827 (+ 27446us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:07:13.900831 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:07:13.900833 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:07:13.900852 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:07:13.902339 (+ 1487us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:07:13.902770 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:07:13.902773 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:07:13.902775 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:07:13.902847 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:07:13.903073 (+ 226us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:07:13.903075 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:07:13.903077 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:07:13.903277 (+ 200us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:07:13.903357 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0620 06:07:13.904425 (+ 1068us) server_negotiation.cc:299] Negotiation successful
0620 06:07:13.904527 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260620 06:07:13.905195 18518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:07:13.869241 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.102.61:39977 (local address 127.0.0.1:52488)
0620 06:07:13.869746 (+ 505us) negotiation.cc:107] Waiting for socket to connect
0620 06:07:13.869758 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 06:07:13.869871 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:07:13.870209 (+ 338us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:07:13.870214 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:07:13.870519 (+ 305us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:07:13.870639 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:07:13.870646 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:07:13.871727 (+ 1081us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:07:13.871731 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:07:13.872779 (+ 1048us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:07:13.872789 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:07:13.873505 (+ 716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:07:13.873511 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:07:13.873675 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:07:13.900494 (+ 26819us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:07:13.900509 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:07:13.900941 (+ 432us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:07:13.903631 (+ 2690us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:07:13.903634 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:07:13.903637 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:07:13.903877 (+ 240us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:07:13.904199 (+ 322us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:07:13.904202 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:07:13.904204 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:07:13.904281 (+ 77us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:07:13.904690 (+ 409us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:07:13.904693 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:07:13.904788 (+ 95us) client_negotiation.cc:769] Sending connection context
0620 06:07:13.904882 (+ 94us) client_negotiation.cc:240] Negotiation successful
0620 06:07:13.905013 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":447,"mutex_wait_us":131,"thread_start_us":109,"threads_started":1}
I20260620 06:07:13.905726 18498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:07:13.869357 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52488 (local address 127.11.102.61:39977)
0620 06:07:13.869460 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0620 06:07:13.869464 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 06:07:13.869818 (+ 354us) server_negotiation.cc:407] Connection header received
0620 06:07:13.869966 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:07:13.869969 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:07:13.870019 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:07:13.870065 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:07:13.870778 (+ 713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:07:13.871571 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:07:13.872913 (+ 1342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:07:13.873285 (+ 372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:07:13.873383 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:07:13.901379 (+ 27996us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:07:13.901384 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:07:13.901387 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:07:13.901417 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:07:13.903489 (+ 2072us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:07:13.904003 (+ 514us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:07:13.904007 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:07:13.904009 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:07:13.904091 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:07:13.904371 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:07:13.904374 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:07:13.904376 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:07:13.904568 (+ 192us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:07:13.904651 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0620 06:07:13.905459 (+ 808us) server_negotiation.cc:299] Negotiation successful
0620 06:07:13.905581 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260620 06:07:13.908159 18331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:07:13.881747 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.102.60:41691 (local address 127.0.0.1:56096)
0620 06:07:13.881923 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0620 06:07:13.881938 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 06:07:13.882009 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:07:13.882298 (+ 289us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:07:13.882304 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:07:13.882503 (+ 199us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:07:13.882640 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:07:13.882650 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:07:13.883688 (+ 1038us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:07:13.883695 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:07:13.884721 (+ 1026us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:07:13.884734 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:07:13.885324 (+ 590us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:07:13.885331 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:07:13.885462 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:07:13.885959 (+ 497us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:07:13.885978 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:07:13.887554 (+ 1576us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:07:13.906489 (+ 18935us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:07:13.906495 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:07:13.906500 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:07:13.906792 (+ 292us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:07:13.907107 (+ 315us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:07:13.907110 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:07:13.907112 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:07:13.907218 (+ 106us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:07:13.907598 (+ 380us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:07:13.907601 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:07:13.907733 (+ 132us) client_negotiation.cc:769] Sending connection context
0620 06:07:13.907844 (+ 111us) client_negotiation.cc:240] Negotiation successful
0620 06:07:13.907995 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":98}
I20260620 06:07:13.908183 18520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:07:13.881810 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56096 (local address 127.11.102.60:41691)
0620 06:07:13.882035 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0620 06:07:13.882042 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 06:07:13.882054 (+ 12us) server_negotiation.cc:407] Connection header received
0620 06:07:13.882114 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:07:13.882116 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:07:13.882173 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:07:13.882262 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:07:13.882758 (+ 496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:07:13.883548 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:07:13.884846 (+ 1298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:07:13.885207 (+ 361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:07:13.885305 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:07:13.887736 (+ 2431us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:07:13.887743 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:07:13.887749 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:07:13.887780 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:07:13.906344 (+ 18564us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:07:13.906917 (+ 573us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:07:13.906920 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:07:13.906922 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:07:13.906999 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:07:13.907318 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:07:13.907321 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:07:13.907323 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:07:13.907500 (+ 177us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:07:13.907588 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0620 06:07:13.907921 (+ 333us) server_negotiation.cc:299] Negotiation successful
0620 06:07:13.908028 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":173,"thread_start_us":94,"threads_started":1}
I20260620 06:07:13.910156 17774 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.11.102.60:41691: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "ec929d712e20476684cecae76f307d8e"
member_type: NON_VOTER
last_known_addr {
host: "127.11.102.60"
port: 41691
}
attrs {
promote: true
}
}
dest_uuid: "d7d4500dc38a48a1858293d72d1fcfc2"
cas_config_opid_index: 5
I20260620 06:07:13.911132 17774 consensus_queue.cc:237] T 00000000000000000000000000000000 P d7d4500dc38a48a1858293d72d1fcfc2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "d7d4500dc38a48a1858293d72d1fcfc2" member_type: VOTER last_known_addr { host: "127.11.102.62" port: 44777 } } peers { permanent_uuid: "0a9e7156fd5c4d399486dc5932b407b2" member_type: NON_VOTER last_known_addr { host: "127.11.102.61" port: 39977 } attrs { promote: true } } peers { permanent_uuid: "ec929d712e20476684cecae76f307d8e" member_type: NON_VOTER last_known_addr { host: "127.11.102.60" port: 41691 } attrs { promote: true } }
I20260620 06:07:13.912402 18335 raft_consensus.cc:2932] T 00000000000000000000000000000000 P d7d4500dc38a48a1858293d72d1fcfc2 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER ec929d712e20476684cecae76f307d8e (127.11.102.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "d7d4500dc38a48a1858293d72d1fcfc2" member_type: VOTER last_known_addr { host: "127.11.102.62" port: 44777 } } peers { permanent_uuid: "0a9e7156fd5c4d399486dc5932b407b2" member_type: NON_VOTER last_known_addr { host: "127.11.102.61" port: 39977 } attrs { promote: true } } peers { permanent_uuid: "ec929d712e20476684cecae76f307d8e" member_type: NON_VOTER last_known_addr { host: "127.11.102.60" port: 41691 } attrs { promote: true } } }
I20260620 06:07:13.913259 18468 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ec929d712e20476684cecae76f307d8e [term 0 FOLLOWER]: Advancing to term 1
I20260620 06:07:13.913802 18335 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.11.102.60:41691
I20260620 06:07:13.913980 18335 sys_catalog.cc:455] T 00000000000000000000000000000000 P d7d4500dc38a48a1858293d72d1fcfc2 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "d7d4500dc38a48a1858293d72d1fcfc2" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "d7d4500dc38a48a1858293d72d1fcfc2" member_type: VOTER last_known_addr { host: "127.11.102.62" port: 44777 } } peers { permanent_uuid: "0a9e7156fd5c4d399486dc5932b407b2" member_type: NON_VOTER last_known_addr { host: "127.11.102.61" port: 39977 } attrs { promote: true } } peers { permanent_uuid: "ec929d712e20476684cecae76f307d8e" member_type: NON_VOTER last_known_addr { host: "127.11.102.60" port: 41691 } attrs { promote: true } } }
I20260620 06:07:13.914067 18335 sys_catalog.cc:458] T 00000000000000000000000000000000 P d7d4500dc38a48a1858293d72d1fcfc2 [sys.catalog]: This master's current role is: LEADER
I20260620 06:07:13.914482 18346 master.cc:537] Master@127.11.102.60:41691 shutting down...
I20260620 06:07:13.914613 18468 raft_consensus.cc:1261] T 00000000000000000000000000000000 P ec929d712e20476684cecae76f307d8e [term 1 FOLLOWER]: Refusing update from remote peer d7d4500dc38a48a1858293d72d1fcfc2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260620 06:07:13.915306 18336 consensus_queue.cc:1035] T 00000000000000000000000000000000 P d7d4500dc38a48a1858293d72d1fcfc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec929d712e20476684cecae76f307d8e" member_type: NON_VOTER last_known_addr { host: "127.11.102.60" port: 41691 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.000s
W20260620 06:07:13.917443 17759 consensus_peers.cc:487] T 00000000000000000000000000000000 P d7d4500dc38a48a1858293d72d1fcfc2 -> Peer ec929d712e20476684cecae76f307d8e (127.11.102.60:41691): Couldn't send request to peer ec929d712e20476684cecae76f307d8e. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260620 06:07:13.919322 18346 raft_consensus.cc:2229] T 00000000000000000000000000000000 P ec929d712e20476684cecae76f307d8e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 06:07:13.919585 18346 raft_consensus.cc:2258] T 00000000000000000000000000000000 P ec929d712e20476684cecae76f307d8e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 06:07:13.919725 18346 tablet_replica.cc:330] T 00000000000000000000000000000000 P ec929d712e20476684cecae76f307d8e: stopping tablet replica
I20260620 06:07:13.920987 18331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:07:13.912447 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.102.61:39977 (local address 127.0.0.1:52496)
0620 06:07:13.915262 (+ 2815us) negotiation.cc:107] Waiting for socket to connect
0620 06:07:13.915277 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 06:07:13.915497 (+ 220us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:07:13.916691 (+ 1194us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:07:13.916694 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:07:13.916710 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 06:07:13.916790 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:07:13.916797 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:07:13.917876 (+ 1079us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:07:13.917879 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:07:13.919332 (+ 1453us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:07:13.919343 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:07:13.920606 (+ 1263us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:07:13.920610 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:07:13.920695 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:07:13.920735 (+ 40us) client_negotiation.cc:769] Sending connection context
0620 06:07:13.920807 (+ 72us) client_negotiation.cc:240] Negotiation successful
0620 06:07:13.920873 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2745}
I20260620 06:07:13.921373 18498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:07:13.915222 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52496 (local address 127.11.102.61:39977)
0620 06:07:13.916364 (+ 1142us) server_negotiation.cc:206] Beginning negotiation
0620 06:07:13.916368 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 06:07:13.916381 (+ 13us) server_negotiation.cc:407] Connection header received
0620 06:07:13.916422 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:07:13.916425 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:07:13.916478 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:07:13.916562 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 06:07:13.916914 (+ 352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:07:13.917747 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:07:13.919548 (+ 1801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:07:13.920299 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:07:13.920397 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:07:13.920530 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0620 06:07:13.921201 (+ 671us) server_negotiation.cc:299] Negotiation successful
0620 06:07:13.921270 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1066}
I20260620 06:07:13.922148 18380 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 0a9e7156fd5c4d399486dc5932b407b2 [term 1 LEARNER]: Refusing update from remote peer d7d4500dc38a48a1858293d72d1fcfc2: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260620 06:07:13.922698 18336 consensus_queue.cc:1035] T 00000000000000000000000000000000 P d7d4500dc38a48a1858293d72d1fcfc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a9e7156fd5c4d399486dc5932b407b2" member_type: NON_VOTER last_known_addr { host: "127.11.102.61" port: 39977 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260620 06:07:13.924655 18380 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 0a9e7156fd5c4d399486dc5932b407b2 [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER ec929d712e20476684cecae76f307d8e (127.11.102.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "d7d4500dc38a48a1858293d72d1fcfc2" member_type: VOTER last_known_addr { host: "127.11.102.62" port: 44777 } } peers { permanent_uuid: "0a9e7156fd5c4d399486dc5932b407b2" member_type: NON_VOTER last_known_addr { host: "127.11.102.61" port: 39977 } attrs { promote: true } } peers { permanent_uuid: "ec929d712e20476684cecae76f307d8e" member_type: NON_VOTER last_known_addr { host: "127.11.102.60" port: 41691 } attrs { promote: true } } }
I20260620 06:07:13.926472 18523 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0a9e7156fd5c4d399486dc5932b407b2 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "d7d4500dc38a48a1858293d72d1fcfc2" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "d7d4500dc38a48a1858293d72d1fcfc2" member_type: VOTER last_known_addr { host: "127.11.102.62" port: 44777 } } peers { permanent_uuid: "0a9e7156fd5c4d399486dc5932b407b2" member_type: NON_VOTER last_known_addr { host: "127.11.102.61" port: 39977 } attrs { promote: true } } peers { permanent_uuid: "ec929d712e20476684cecae76f307d8e" member_type: NON_VOTER last_known_addr { host: "127.11.102.60" port: 41691 } attrs { promote: true } } }
I20260620 06:07:13.926652 18523 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0a9e7156fd5c4d399486dc5932b407b2 [sys.catalog]: This master's current role is: LEARNER
I20260620 06:07:13.926827 18415 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0a9e7156fd5c4d399486dc5932b407b2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d7d4500dc38a48a1858293d72d1fcfc2. Latest consensus state: current_term: 1 leader_uuid: "d7d4500dc38a48a1858293d72d1fcfc2" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "d7d4500dc38a48a1858293d72d1fcfc2" member_type: VOTER last_known_addr { host: "127.11.102.62" port: 44777 } } peers { permanent_uuid: "0a9e7156fd5c4d399486dc5932b407b2" member_type: NON_VOTER last_known_addr { host: "127.11.102.61" port: 39977 } attrs { promote: true } } peers { permanent_uuid: "ec929d712e20476684cecae76f307d8e" member_type: NON_VOTER last_known_addr { host: "127.11.102.60" port: 41691 } attrs { promote: true } } }
I20260620 06:07:13.926942 18415 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0a9e7156fd5c4d399486dc5932b407b2 [sys.catalog]: This master's current role is: LEARNER
I20260620 06:07:13.927544 18336 raft_consensus.cc:1050] T 00000000000000000000000000000000 P d7d4500dc38a48a1858293d72d1fcfc2: attempting to promote NON_VOTER 0a9e7156fd5c4d399486dc5932b407b2 to VOTER
I20260620 06:07:13.927994 18336 consensus_queue.cc:237] T 00000000000000000000000000000000 P d7d4500dc38a48a1858293d72d1fcfc2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "d7d4500dc38a48a1858293d72d1fcfc2" member_type: VOTER last_known_addr { host: "127.11.102.62" port: 44777 } } peers { permanent_uuid: "0a9e7156fd5c4d399486dc5932b407b2" member_type: VOTER last_known_addr { host: "127.11.102.61" port: 39977 } attrs { promote: false } } peers { permanent_uuid: "ec929d712e20476684cecae76f307d8e" member_type: NON_VOTER last_known_addr { host: "127.11.102.60" port: 41691 } attrs { promote: true } }
I20260620 06:07:13.929001 18380 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 0a9e7156fd5c4d399486dc5932b407b2 [term 1 LEARNER]: Refusing update from remote peer d7d4500dc38a48a1858293d72d1fcfc2: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260620 06:07:13.929033 18331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:07:13.928815 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.102.60:41691 (local address 127.0.0.1:56104)
0620 06:07:13.928910 (+ 95us) negotiation.cc:107] Waiting for socket to connect
0620 06:07:13.928967 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.102.60:41691: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
I20260620 06:07:13.929414 18335 consensus_queue.cc:1035] T 00000000000000000000000000000000 P d7d4500dc38a48a1858293d72d1fcfc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a9e7156fd5c4d399486dc5932b407b2" member_type: VOTER last_known_addr { host: "127.11.102.61" port: 39977 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260620 06:07:13.929700 18331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:07:13.929482 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.102.60:41691 (local address 127.0.0.1:56108)
0620 06:07:13.929598 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0620 06:07:13.929641 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.102.60:41691: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
W20260620 06:07:13.929801 17759 consensus_peers.cc:487] T 00000000000000000000000000000000 P d7d4500dc38a48a1858293d72d1fcfc2 -> Peer ec929d712e20476684cecae76f307d8e (127.11.102.60:41691): Couldn't send request to peer ec929d712e20476684cecae76f307d8e. Status: Network error: Client connection negotiation failed: client connection to 127.11.102.60:41691: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260620 06:07:13.931154 18332 raft_consensus.cc:2932] T 00000000000000000000000000000000 P d7d4500dc38a48a1858293d72d1fcfc2 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 0a9e7156fd5c4d399486dc5932b407b2 (127.11.102.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "d7d4500dc38a48a1858293d72d1fcfc2" member_type: VOTER last_known_addr { host: "127.11.102.62" port: 44777 } } peers { permanent_uuid: "0a9e7156fd5c4d399486dc5932b407b2" member_type: VOTER last_known_addr { host: "127.11.102.61" port: 39977 } attrs { promote: false } } peers { permanent_uuid: "ec929d712e20476684cecae76f307d8e" member_type: NON_VOTER last_known_addr { host: "127.11.102.60" port: 41691 } attrs { promote: true } } }
I20260620 06:07:13.931643 18381 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 0a9e7156fd5c4d399486dc5932b407b2 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 0a9e7156fd5c4d399486dc5932b407b2 (127.11.102.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "d7d4500dc38a48a1858293d72d1fcfc2" member_type: VOTER last_known_addr { host: "127.11.102.62" port: 44777 } } peers { permanent_uuid: "0a9e7156fd5c4d399486dc5932b407b2" member_type: VOTER last_known_addr { host: "127.11.102.61" port: 39977 } attrs { promote: false } } peers { permanent_uuid: "ec929d712e20476684cecae76f307d8e" member_type: NON_VOTER last_known_addr { host: "127.11.102.60" port: 41691 } attrs { promote: true } } }
I20260620 06:07:13.932520 18332 sys_catalog.cc:455] T 00000000000000000000000000000000 P d7d4500dc38a48a1858293d72d1fcfc2 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "d7d4500dc38a48a1858293d72d1fcfc2" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "d7d4500dc38a48a1858293d72d1fcfc2" member_type: VOTER last_known_addr { host: "127.11.102.62" port: 44777 } } peers { permanent_uuid: "0a9e7156fd5c4d399486dc5932b407b2" member_type: VOTER last_known_addr { host: "127.11.102.61" port: 39977 } attrs { promote: false } } peers { permanent_uuid: "ec929d712e20476684cecae76f307d8e" member_type: NON_VOTER last_known_addr { host: "127.11.102.60" port: 41691 } attrs { promote: true } } }
I20260620 06:07:13.932610 18332 sys_catalog.cc:458] T 00000000000000000000000000000000 P d7d4500dc38a48a1858293d72d1fcfc2 [sys.catalog]: This master's current role is: LEADER
I20260620 06:07:13.932725 18332 sys_catalog.cc:455] T 00000000000000000000000000000000 P d7d4500dc38a48a1858293d72d1fcfc2 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "d7d4500dc38a48a1858293d72d1fcfc2" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "d7d4500dc38a48a1858293d72d1fcfc2" member_type: VOTER last_known_addr { host: "127.11.102.62" port: 44777 } } peers { permanent_uuid: "0a9e7156fd5c4d399486dc5932b407b2" member_type: VOTER last_known_addr { host: "127.11.102.61" port: 39977 } attrs { promote: false } } peers { permanent_uuid: "ec929d712e20476684cecae76f307d8e" member_type: NON_VOTER last_known_addr { host: "127.11.102.60" port: 41691 } attrs { promote: true } } }
I20260620 06:07:13.932788 18332 sys_catalog.cc:458] T 00000000000000000000000000000000 P d7d4500dc38a48a1858293d72d1fcfc2 [sys.catalog]: This master's current role is: LEADER
I20260620 06:07:13.933259 18415 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0a9e7156fd5c4d399486dc5932b407b2 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "d7d4500dc38a48a1858293d72d1fcfc2" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "d7d4500dc38a48a1858293d72d1fcfc2" member_type: VOTER last_known_addr { host: "127.11.102.62" port: 44777 } } peers { permanent_uuid: "0a9e7156fd5c4d399486dc5932b407b2" member_type: VOTER last_known_addr { host: "127.11.102.61" port: 39977 } attrs { promote: false } } peers { permanent_uuid: "ec929d712e20476684cecae76f307d8e" member_type: NON_VOTER last_known_addr { host: "127.11.102.60" port: 41691 } attrs { promote: true } } }
I20260620 06:07:13.933362 18415 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0a9e7156fd5c4d399486dc5932b407b2 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 06:07:13.935302 18346 master.cc:552] Master@127.11.102.60:41691 shutdown complete.
I20260620 06:07:13.935351 18346 master_runner.cc:305] Clearing existing system tablet
I20260620 06:07:13.935577 18346 fs_manager.cc:359] Metadata directory not provided
I20260620 06:07:13.935680 18346 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:07:13.936929 18346 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260620 06:07:13.937810 18525 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:07:13.938042 18346 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935603120653-11672-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:07:13.938170 18346 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 06:07:13.938205 18346 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935603120653-11672-0/minicluster-data/master-2/data,/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935603120653-11672-0/minicluster-data/master-2/wal
uuid: "ec929d712e20476684cecae76f307d8e"
format_stamp: "Formatted at 2026-06-20 06:07:13 on dist-test-slave-9hm5"
server_key: "8ccea1aa0515b1fbeea42eeee4193cc7"
server_key_iv: "7dd9d3a44a97356d9d87a40c9a72f8e2"
server_key_version: "encryptionkey@0"
I20260620 06:07:13.940163 18346 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P ec929d712e20476684cecae76f307d8e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 06:07:13.942387 18346 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P ec929d712e20476684cecae76f307d8e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260620 06:07:13.942471 18346 log.cc:1196] T 00000000000000000000000000000000 P ec929d712e20476684cecae76f307d8e: Deleting WAL directory at /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935603120653-11672-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260620 06:07:13.942708 18346 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P ec929d712e20476684cecae76f307d8e: Deleting consensus metadata
I20260620 06:07:13.942853 18346 master_runner.cc:315] Copying system tablet from 127.11.102.62:44777
I20260620 06:07:13.943954 18346 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P ec929d712e20476684cecae76f307d8e: tablet copy: Beginning tablet copy session from remote peer at address 127.11.102.62:44777
I20260620 06:07:13.959000 17813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:07:13.944547 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54660 (local address 127.11.102.62:44777)
0620 06:07:13.944660 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0620 06:07:13.944665 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 06:07:13.950916 (+ 6251us) server_negotiation.cc:407] Connection header received
0620 06:07:13.951020 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:07:13.951024 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:07:13.951077 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:07:13.951176 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:07:13.951764 (+ 588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:07:13.952583 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:07:13.953705 (+ 1122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:07:13.954047 (+ 342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:07:13.954156 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:07:13.955454 (+ 1298us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:07:13.955458 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:07:13.955461 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:07:13.955487 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:07:13.957216 (+ 1729us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:07:13.957700 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:07:13.957705 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:07:13.957708 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:07:13.957806 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:07:13.958087 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:07:13.958090 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:07:13.958092 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:07:13.958310 (+ 218us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:07:13.958403 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0620 06:07:13.958688 (+ 285us) server_negotiation.cc:299] Negotiation successful
0620 06:07:13.958806 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260620 06:07:13.959008 18531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:07:13.944462 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.102.62:44777 (local address 127.0.0.1:54660)
0620 06:07:13.950789 (+ 6327us) negotiation.cc:107] Waiting for socket to connect
0620 06:07:13.950806 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 06:07:13.950934 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:07:13.951209 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:07:13.951213 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:07:13.951432 (+ 219us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:07:13.951572 (+ 140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:07:13.951590 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:07:13.952703 (+ 1113us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:07:13.952706 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:07:13.953574 (+ 868us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:07:13.953595 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:07:13.954193 (+ 598us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:07:13.954198 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:07:13.954292 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:07:13.954861 (+ 569us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:07:13.954875 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:07:13.955308 (+ 433us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:07:13.957329 (+ 2021us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:07:13.957333 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:07:13.957335 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:07:13.957586 (+ 251us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:07:13.957901 (+ 315us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:07:13.957904 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:07:13.957906 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:07:13.957995 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:07:13.958405 (+ 410us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:07:13.958408 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:07:13.958505 (+ 97us) client_negotiation.cc:769] Sending connection context
0620 06:07:13.958619 (+ 114us) client_negotiation.cc:240] Negotiation successful
0620 06:07:13.958774 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6250,"thread_start_us":82,"threads_started":1}
I20260620 06:07:13.959918 17794 tablet_copy_service.cc:140] P d7d4500dc38a48a1858293d72d1fcfc2: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer ec929d712e20476684cecae76f307d8e ({username='oryx', principal='oryx/127.11.102.60@KRBTEST.COM'} at 127.0.0.1:54660)
I20260620 06:07:13.960012 17794 tablet_copy_service.cc:161] P d7d4500dc38a48a1858293d72d1fcfc2: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer ec929d712e20476684cecae76f307d8e at {username='oryx', principal='oryx/127.11.102.60@KRBTEST.COM'} at 127.0.0.1:54660: session id = ec929d712e20476684cecae76f307d8e-00000000000000000000000000000000
I20260620 06:07:13.961099 17794 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P d7d4500dc38a48a1858293d72d1fcfc2: Tablet Copy: opened 0 blocks and 1 log segments
I20260620 06:07:13.962370 18346 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260620 06:07:13.964692 18346 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P ec929d712e20476684cecae76f307d8e: tablet copy: Starting download of 0 data blocks...
I20260620 06:07:13.964865 18346 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P ec929d712e20476684cecae76f307d8e: tablet copy: Starting download of 1 WAL segments...
I20260620 06:07:13.966344 18346 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P ec929d712e20476684cecae76f307d8e: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260620 06:07:13.968173 17794 tablet_copy_service.cc:342] P d7d4500dc38a48a1858293d72d1fcfc2: Request end of tablet copy session ec929d712e20476684cecae76f307d8e-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.11.102.60@KRBTEST.COM'} at 127.0.0.1:54660
I20260620 06:07:13.968250 17794 tablet_copy_service.cc:433] P d7d4500dc38a48a1858293d72d1fcfc2: ending tablet copy session ec929d712e20476684cecae76f307d8e-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer ec929d712e20476684cecae76f307d8e
I20260620 06:07:13.976120 18346 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 06:07:13.977670 18532 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:07:13.977739 18533 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:07:13.978016 18535 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:07:13.978325 18346 server_base.cc:1034] running on GCE node
I20260620 06:07:13.978494 18346 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:07:13.978816 18346 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:07:13.980007 18346 hybrid_clock.cc:648] HybridClock initialized: now 1781935633979996 us; error 28 us; skew 500 ppm
Jun 20 06:07:13 dist-test-slave-9hm5 krb5kdc[17729](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935633, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.102.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 06:07:13.981670 18346 init.cc:377] Logged in from keytab as oryx/127.11.102.60@KRBTEST.COM (short username oryx)
I20260620 06:07:13.982137 18346 webserver.cc:460] Webserver started at http://127.11.102.60:42919/ using document root <none> and password file <none>
I20260620 06:07:13.982357 18346 fs_manager.cc:359] Metadata directory not provided
I20260620 06:07:13.982434 18346 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:07:13.983795 18346 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 06:07:13.984768 18542 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:07:13.985025 18346 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 06:07:13.985100 18346 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935603120653-11672-0/minicluster-data/master-2/data,/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935603120653-11672-0/minicluster-data/master-2/wal
uuid: "ec929d712e20476684cecae76f307d8e"
format_stamp: "Formatted at 2026-06-20 06:07:13 on dist-test-slave-9hm5"
server_key: "8ccea1aa0515b1fbeea42eeee4193cc7"
server_key_iv: "7dd9d3a44a97356d9d87a40c9a72f8e2"
server_key_version: "encryptionkey@0"
I20260620 06:07:13.985169 18346 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935603120653-11672-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935603120653-11672-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935603120653-11672-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
W20260620 06:07:14.020421 18521 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:07:14.020664 18521 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:07:14.020735 18521 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:07:14.024597 18521 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:07:14.024663 18521 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:07:14.024768 18521 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.102.1
I20260620 06:07:14.029798 18521 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.102.20:34945
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935603120653-11672-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935603120653-11672-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935603120653-11672-0/krb5kdc/oryx_127.11.102.1.keytab
--principal=oryx/127.11.102.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.102.1:37681
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935603120653-11672-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.11.102.1
--webserver_port=36757
--tserver_master_addrs=127.11.102.62:44777,127.11.102.61:39977,127.11.102.60:41691
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18521
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.102.1
--log_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935603120653-11672-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 05:28:48 UTC on ccycloud.quasar-fixhms.root.comops.site
build id 4457020
I20260620 06:07:14.030783 18521 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:07:14.031824 18521 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 06:07:14.032843 18346 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:07:14.034989 18346 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:07:14.040700 18346 rpc_server.cc:307] RPC server started. Bound to: 127.11.102.60:41691
I20260620 06:07:14.040793 18602 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.102.60:41691 every 8 connection(s)
I20260620 06:07:14.041319 18346 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935603120653-11672-0/minicluster-data/master-2/data/info.pb
W20260620 06:07:14.042953 18560 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:07:14.042953 18554 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:07:14.043058 18557 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:07:14.043428 18521 server_base.cc:1034] running on GCE node
I20260620 06:07:14.043857 18521 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:07:14.044232 18603 sys_catalog.cc:263] Verifying existing consensus state
I20260620 06:07:14.044565 18521 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:07:14.045212 18603 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ec929d712e20476684cecae76f307d8e: Bootstrap starting.
I20260620 06:07:14.045750 18521 hybrid_clock.cc:648] HybridClock initialized: now 1781935634045728 us; error 35 us; skew 500 ppm
Jun 20 06:07:14 dist-test-slave-9hm5 krb5kdc[17729](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935634, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.102.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 06:07:14.049149 18521 init.cc:377] Logged in from keytab as oryx/127.11.102.1@KRBTEST.COM (short username oryx)
I20260620 06:07:14.050101 18521 webserver.cc:460] Webserver started at http://127.11.102.1:36757/ using document root <none> and password file <none>
I20260620 06:07:14.050758 18521 fs_manager.cc:359] Metadata directory not provided
I20260620 06:07:14.050850 18521 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:07:14.054755 18521 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260620 06:07:14.054921 18603 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ec929d712e20476684cecae76f307d8e: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 06:07:14.055169 18603 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ec929d712e20476684cecae76f307d8e: Bootstrap complete.
I20260620 06:07:14.055689 18603 raft_consensus.cc:348] T 00000000000000000000000000000000 P ec929d712e20476684cecae76f307d8e [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "d7d4500dc38a48a1858293d72d1fcfc2" member_type: VOTER last_known_addr { host: "127.11.102.62" port: 44777 } } peers { permanent_uuid: "0a9e7156fd5c4d399486dc5932b407b2" member_type: VOTER last_known_addr { host: "127.11.102.61" port: 39977 } attrs { promote: false } } peers { permanent_uuid: "ec929d712e20476684cecae76f307d8e" member_type: NON_VOTER last_known_addr { host: "127.11.102.60" port: 41691 } attrs { promote: true } }
I20260620 06:07:14.055816 18603 raft_consensus.cc:729] T 00000000000000000000000000000000 P ec929d712e20476684cecae76f307d8e [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: ec929d712e20476684cecae76f307d8e, State: Initialized, Role: LEARNER
I20260620 06:07:14.055948 18603 consensus_queue.cc:260] T 00000000000000000000000000000000 P ec929d712e20476684cecae76f307d8e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "d7d4500dc38a48a1858293d72d1fcfc2" member_type: VOTER last_known_addr { host: "127.11.102.62" port: 44777 } } peers { permanent_uuid: "0a9e7156fd5c4d399486dc5932b407b2" member_type: VOTER last_known_addr { host: "127.11.102.61" port: 39977 } attrs { promote: false } } peers { permanent_uuid: "ec929d712e20476684cecae76f307d8e" member_type: NON_VOTER last_known_addr { host: "127.11.102.60" port: 41691 } attrs { promote: true } }
I20260620 06:07:14.056356 18603 sys_catalog.cc:564] T 00000000000000000000000000000000 P ec929d712e20476684cecae76f307d8e [sys.catalog]: configured and running, proceeding with master startup.
I20260620 06:07:14.058075 18612 sys_catalog.cc:455] T 00000000000000000000000000000000 P ec929d712e20476684cecae76f307d8e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "d7d4500dc38a48a1858293d72d1fcfc2" member_type: VOTER last_known_addr { host: "127.11.102.62" port: 44777 } } peers { permanent_uuid: "0a9e7156fd5c4d399486dc5932b407b2" member_type: VOTER last_known_addr { host: "127.11.102.61" port: 39977 } attrs { promote: false } } peers { permanent_uuid: "ec929d712e20476684cecae76f307d8e" member_type: NON_VOTER last_known_addr { host: "127.11.102.60" port: 41691 } attrs { promote: true } } }
I20260620 06:07:14.059991 18612 sys_catalog.cc:458] T 00000000000000000000000000000000 P ec929d712e20476684cecae76f307d8e [sys.catalog]: This master's current role is: LEARNER
I20260620 06:07:14.058797 18623 catalog_manager.cc:1259] Loaded cluster ID: bcc977c4637644dab45ae102d36c03ef
I20260620 06:07:14.060086 18623 catalog_manager.cc:1552] T 00000000000000000000000000000000 P ec929d712e20476684cecae76f307d8e: loading cluster ID for follower catalog manager: success
I20260620 06:07:14.060837 18624 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:07:14.062266 18521 fs_manager.cc:725] Time spent opening block manager: real 0.006s user 0.001s sys 0.000s
I20260620 06:07:14.062408 18623 catalog_manager.cc:1574] T 00000000000000000000000000000000 P ec929d712e20476684cecae76f307d8e: acquiring CA information for follower catalog manager: success
I20260620 06:07:14.062395 18521 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935603120653-11672-0/minicluster-data/ts-0/data,/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935603120653-11672-0/minicluster-data/ts-0/wal
uuid: "fb0783328ad3442fba39f260adf25308"
format_stamp: "Formatted at 2026-06-20 06:07:12 on dist-test-slave-9hm5"
server_key: "c256682846b1d9a11755b37708093731"
server_key_iv: "5e598148fb006f306cd6f4985ee7540d"
server_key_version: "encryptionkey@0"
I20260620 06:07:14.062981 18521 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935603120653-11672-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935603120653-11672-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935603120653-11672-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:07:14.063647 18623 catalog_manager.cc:1602] T 00000000000000000000000000000000 P ec929d712e20476684cecae76f307d8e: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 06:07:14.076327 18521 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:07:14.078284 18099 heartbeater.cc:503] Master 127.11.102.62:44777 was elected leader, sending a full tablet report...
I20260620 06:07:14.079798 18521 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:07:14.079948 18521 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:07:14.080442 18521 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 06:07:14.081212 18521 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 06:07:14.081267 18521 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 06:07:14.081305 18521 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 06:07:14.081321 18521 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 06:07:14.089975 18521 rpc_server.cc:307] RPC server started. Bound to: 127.11.102.1:37681
I20260620 06:07:14.090067 18737 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.102.1:37681 every 8 connection(s)
I20260620 06:07:14.090997 18521 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935603120653-11672-0/minicluster-data/ts-0/data/info.pb
I20260620 06:07:14.098091 11672 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9HMp43/build/debug/bin/kudu as pid 18521
I20260620 06:07:14.098271 11672 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9HMp43/build/debug/bin/kudu with pid 17970
Jun 20 06:07:14 dist-test-slave-9hm5 krb5kdc[17729](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935634, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.102.1@KRBTEST.COM for oryx/127.11.102.61@KRBTEST.COM
Jun 20 06:07:14 dist-test-slave-9hm5 krb5kdc[17729](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935634, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.102.1@KRBTEST.COM for oryx/127.11.102.60@KRBTEST.COM
Jun 20 06:07:14 dist-test-slave-9hm5 krb5kdc[17729](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935634, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.102.1@KRBTEST.COM for oryx/127.11.102.62@KRBTEST.COM
I20260620 06:07:14.106189 11672 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9HMp43/build/debug/bin/kudu
/tmp/dist-test-task9HMp43/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935603120653-11672-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935603120653-11672-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935603120653-11672-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935603120653-11672-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.102.2:33643
--local_ip_for_outbound_sockets=127.11.102.2
--tserver_master_addrs=127.11.102.62:44777
--webserver_port=34063
--webserver_interface=127.11.102.2
--keytab_file=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935603120653-11672-0/krb5kdc/oryx_127.11.102.2.keytab
--principal=oryx/127.11.102.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.102.20:34945
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.11.102.62:44777,127.11.102.61:39977
--tserver_master_addrs=127.11.102.62:44777,127.11.102.61:39977,127.11.102.60:41691 with env {KRB5CCNAME=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935603120653-11672-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935603120653-11672-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935603120653-11672-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 06:07:14.108942 18498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:07:14.092834 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.102.1:34205 (local address 127.11.102.61:39977)
0620 06:07:14.092958 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0620 06:07:14.092962 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 06:07:14.094121 (+ 1159us) server_negotiation.cc:407] Connection header received
0620 06:07:14.095333 (+ 1212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:07:14.095337 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:07:14.095407 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:07:14.095453 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:07:14.097388 (+ 1935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:07:14.098184 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:07:14.100331 (+ 2147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:07:14.100663 (+ 332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:07:14.100729 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:07:14.104310 (+ 3581us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:07:14.104315 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:07:14.104320 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:07:14.104374 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:07:14.106389 (+ 2015us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:07:14.107059 (+ 670us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:07:14.107062 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:07:14.107065 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:07:14.107155 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:07:14.107588 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:07:14.107592 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:07:14.107594 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:07:14.107856 (+ 262us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:07:14.107938 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0620 06:07:14.108633 (+ 695us) server_negotiation.cc:299] Negotiation successful
0620 06:07:14.108754 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260620 06:07:14.109455 17813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:07:14.092834 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.102.1:53413 (local address 127.11.102.62:44777)
0620 06:07:14.092957 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0620 06:07:14.092962 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 06:07:14.094121 (+ 1159us) server_negotiation.cc:407] Connection header received
0620 06:07:14.095353 (+ 1232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:07:14.095357 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:07:14.095414 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:07:14.095498 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:07:14.097392 (+ 1894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:07:14.098191 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:07:14.101492 (+ 3301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:07:14.101802 (+ 310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:07:14.101910 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:07:14.105673 (+ 3763us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:07:14.105676 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:07:14.105678 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:07:14.105710 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:07:14.107741 (+ 2031us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:07:14.108193 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:07:14.108197 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:07:14.108199 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:07:14.108279 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:07:14.108581 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:07:14.108584 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:07:14.108586 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:07:14.108792 (+ 206us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:07:14.108878 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0620 06:07:14.109219 (+ 341us) server_negotiation.cc:299] Negotiation successful
0620 06:07:14.109321 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260620 06:07:14.109586 18745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:07:14.094549 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.102.1:38241 (local address 127.11.102.60:41691)
0620 06:07:14.094784 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0620 06:07:14.094794 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 06:07:14.094807 (+ 13us) server_negotiation.cc:407] Connection header received
0620 06:07:14.095396 (+ 589us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:07:14.095400 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:07:14.095452 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:07:14.095572 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:07:14.097376 (+ 1804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:07:14.098960 (+ 1584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:07:14.100244 (+ 1284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:07:14.100597 (+ 353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:07:14.100709 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:07:14.104769 (+ 4060us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:07:14.104776 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:07:14.104782 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:07:14.104814 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:07:14.107014 (+ 2200us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:07:14.107555 (+ 541us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:07:14.107560 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:07:14.107563 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:07:14.107700 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:07:14.108008 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:07:14.108012 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:07:14.108014 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:07:14.108241 (+ 227us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:07:14.108332 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0620 06:07:14.109323 (+ 991us) server_negotiation.cc:299] Negotiation successful
0620 06:07:14.109421 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":173,"thread_start_us":69,"threads_started":1}
I20260620 06:07:14.110169 18744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:07:14.094453 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.102.60:41691 (local address 127.11.102.1:38241)
0620 06:07:14.094723 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0620 06:07:14.094736 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 06:07:14.095112 (+ 376us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:07:14.095740 (+ 628us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:07:14.095743 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:07:14.096636 (+ 893us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:07:14.097232 (+ 596us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:07:14.097246 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:07:14.099093 (+ 1847us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:07:14.099099 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:07:14.100108 (+ 1009us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:07:14.100119 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:07:14.100755 (+ 636us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:07:14.100762 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:07:14.100988 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:07:14.101857 (+ 869us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:07:14.101877 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:07:14.104563 (+ 2686us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:07:14.107135 (+ 2572us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:07:14.107141 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:07:14.107146 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:07:14.107414 (+ 268us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:07:14.107812 (+ 398us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:07:14.107816 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:07:14.107819 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:07:14.107908 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:07:14.108377 (+ 469us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:07:14.108382 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:07:14.108479 (+ 97us) client_negotiation.cc:769] Sending connection context
0620 06:07:14.108618 (+ 139us) client_negotiation.cc:240] Negotiation successful
0620 06:07:14.108967 (+ 349us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":211,"mutex_wait_us":82,"spinlock_wait_cycles":19328,"thread_start_us":93,"threads_started":1}
I20260620 06:07:14.110189 18742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:07:14.093393 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.102.62:44777 (local address 127.11.102.1:53413)
0620 06:07:14.093961 (+ 568us) negotiation.cc:107] Waiting for socket to connect
0620 06:07:14.094028 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0620 06:07:14.095102 (+ 1074us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:07:14.095715 (+ 613us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:07:14.095725 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:07:14.096644 (+ 919us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:07:14.097244 (+ 600us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:07:14.097251 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:07:14.098346 (+ 1095us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:07:14.098352 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:07:14.099871 (+ 1519us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:07:14.099884 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:07:14.102054 (+ 2170us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:07:14.102059 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:07:14.102178 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:07:14.102667 (+ 489us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:07:14.102687 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:07:14.105515 (+ 2828us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:07:14.107854 (+ 2339us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:07:14.107857 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:07:14.107859 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:07:14.108088 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:07:14.108377 (+ 289us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:07:14.108379 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:07:14.108381 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:07:14.108464 (+ 83us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:07:14.108921 (+ 457us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:07:14.108923 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:07:14.109014 (+ 91us) client_negotiation.cc:769] Sending connection context
0620 06:07:14.109105 (+ 91us) client_negotiation.cc:240] Negotiation successful
0620 06:07:14.109257 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":341,"mutex_wait_us":133,"spinlock_wait_cycles":12416,"thread_start_us":132,"threads_started":1}
I20260620 06:07:14.111598 18738 heartbeater.cc:347] Connected to a master server at 127.11.102.60:41691
I20260620 06:07:14.111857 18739 heartbeater.cc:347] Connected to a master server at 127.11.102.62:44777
I20260620 06:07:14.111863 18738 heartbeater.cc:464] Registering TS with master...
I20260620 06:07:14.111943 18739 heartbeater.cc:464] Registering TS with master...
I20260620 06:07:14.112442 18739 heartbeater.cc:511] Master 127.11.102.62:44777 requested a full tablet report, sending...
I20260620 06:07:14.112458 18738 heartbeater.cc:511] Master 127.11.102.60:41691 requested a full tablet report, sending...
I20260620 06:07:14.113739 17774 ts_manager.cc:194] Re-registered known tserver with Master: fb0783328ad3442fba39f260adf25308 (127.11.102.1:37681)
I20260620 06:07:14.113905 18567 ts_manager.cc:194] Registered new tserver with Master: fb0783328ad3442fba39f260adf25308 (127.11.102.1:37681)
I20260620 06:07:14.114624 17774 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.11.102.1@KRBTEST.COM'} at 127.11.102.1:53413
I20260620 06:07:14.119925 18743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:07:14.093393 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.102.61:39977 (local address 127.11.102.1:34205)
0620 06:07:14.093961 (+ 568us) negotiation.cc:107] Waiting for socket to connect
0620 06:07:14.094019 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0620 06:07:14.095102 (+ 1083us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:07:14.095715 (+ 613us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:07:14.095725 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:07:14.096644 (+ 919us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:07:14.097243 (+ 599us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:07:14.097251 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:07:14.099362 (+ 2111us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:07:14.099365 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:07:14.100218 (+ 853us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:07:14.100225 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:07:14.100871 (+ 646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:07:14.100873 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:07:14.100988 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:07:14.101819 (+ 831us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:07:14.101843 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:07:14.104103 (+ 2260us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:07:14.106532 (+ 2429us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:07:14.106539 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:07:14.106555 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:07:14.106921 (+ 366us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:07:14.107270 (+ 349us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:07:14.107273 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:07:14.107275 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:07:14.107468 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:07:14.108043 (+ 575us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:07:14.108052 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:07:14.108362 (+ 310us) client_negotiation.cc:769] Sending connection context
0620 06:07:14.109108 (+ 746us) client_negotiation.cc:240] Negotiation successful
0620 06:07:14.119794 (+ 10686us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":426,"mutex_wait_us":260,"spinlock_wait_cycles":43136,"thread_start_us":143,"threads_started":1}
I20260620 06:07:14.120618 18740 heartbeater.cc:347] Connected to a master server at 127.11.102.61:39977
I20260620 06:07:14.120707 18740 heartbeater.cc:464] Registering TS with master...
I20260620 06:07:14.120889 18740 heartbeater.cc:511] Master 127.11.102.61:39977 requested a full tablet report, sending...
I20260620 06:07:14.121848 18367 ts_manager.cc:194] Registered new tserver with Master: fb0783328ad3442fba39f260adf25308 (127.11.102.1:37681)
W20260620 06:07:14.222849 18746 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:07:14.223101 18746 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:07:14.223145 18746 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:07:14.227226 18746 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:07:14.227284 18746 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:07:14.227371 18746 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.102.2
I20260620 06:07:14.232576 18746 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.102.20:34945
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935603120653-11672-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935603120653-11672-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935603120653-11672-0/krb5kdc/oryx_127.11.102.2.keytab
--principal=oryx/127.11.102.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.102.2:33643
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935603120653-11672-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.11.102.2
--webserver_port=34063
--tserver_master_addrs=127.11.102.62:44777,127.11.102.61:39977,127.11.102.60:41691
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18746
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.102.2
--log_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935603120653-11672-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 05:28:48 UTC on ccycloud.quasar-fixhms.root.comops.site
build id 4457020
I20260620 06:07:14.233640 18746 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:07:14.234694 18746 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 06:07:14.245045 18752 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:07:14.245115 18751 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:07:14.245121 18754 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:07:14.245327 18746 server_base.cc:1034] running on GCE node
I20260620 06:07:14.245709 18746 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:07:14.246333 18746 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:07:14.247519 18746 hybrid_clock.cc:648] HybridClock initialized: now 1781935634247489 us; error 50 us; skew 500 ppm
Jun 20 06:07:14 dist-test-slave-9hm5 krb5kdc[17729](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935634, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.102.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 06:07:14.250216 18746 init.cc:377] Logged in from keytab as oryx/127.11.102.2@KRBTEST.COM (short username oryx)
I20260620 06:07:14.251099 18746 webserver.cc:460] Webserver started at http://127.11.102.2:34063/ using document root <none> and password file <none>
I20260620 06:07:14.251716 18746 fs_manager.cc:359] Metadata directory not provided
I20260620 06:07:14.251819 18746 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:07:14.255410 18746 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260620 06:07:14.257678 18761 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:07:14.258985 18746 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 06:07:14.259105 18746 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935603120653-11672-0/minicluster-data/ts-1/data,/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935603120653-11672-0/minicluster-data/ts-1/wal
uuid: "f569ba0a0ea74d078dabe54eb42b9e21"
format_stamp: "Formatted at 2026-06-20 06:07:12 on dist-test-slave-9hm5"
server_key: "8bc93c39640867ba8b46e2563ae2aa8e"
server_key_iv: "b8ea8111e3e5e863cbea9878ae36dd87"
server_key_version: "encryptionkey@0"
I20260620 06:07:14.259665 18746 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935603120653-11672-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935603120653-11672-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935603120653-11672-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:07:14.274899 18746 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:07:14.278205 18746 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:07:14.278398 18746 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:07:14.279008 18746 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 06:07:14.279837 18746 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 06:07:14.279881 18746 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 06:07:14.279947 18746 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 06:07:14.279978 18746 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 06:07:14.290153 18746 rpc_server.cc:307] RPC server started. Bound to: 127.11.102.2:33643
I20260620 06:07:14.290182 18874 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.102.2:33643 every 8 connection(s)
I20260620 06:07:14.291086 18746 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935603120653-11672-0/minicluster-data/ts-1/data/info.pb
I20260620 06:07:14.294610 11672 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9HMp43/build/debug/bin/kudu as pid 18746
I20260620 06:07:14.294757 11672 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9HMp43/build/debug/bin/kudu with pid 18104
I20260620 06:07:14.301890 11672 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9HMp43/build/debug/bin/kudu
/tmp/dist-test-task9HMp43/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935603120653-11672-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935603120653-11672-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935603120653-11672-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935603120653-11672-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.102.3:36875
--local_ip_for_outbound_sockets=127.11.102.3
--tserver_master_addrs=127.11.102.62:44777
--webserver_port=43289
--webserver_interface=127.11.102.3
--keytab_file=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935603120653-11672-0/krb5kdc/oryx_127.11.102.3.keytab
--principal=oryx/127.11.102.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.102.20:34945
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.11.102.62:44777,127.11.102.61:39977
--tserver_master_addrs=127.11.102.62:44777,127.11.102.61:39977,127.11.102.60:41691 with env {KRB5CCNAME=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935603120653-11672-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935603120653-11672-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935603120653-11672-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 06:07:14 dist-test-slave-9hm5 krb5kdc[17729](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935634, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.102.2@KRBTEST.COM for oryx/127.11.102.60@KRBTEST.COM
Jun 20 06:07:14 dist-test-slave-9hm5 krb5kdc[17729](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935634, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.102.2@KRBTEST.COM for oryx/127.11.102.61@KRBTEST.COM
Jun 20 06:07:14 dist-test-slave-9hm5 krb5kdc[17729](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935634, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.102.2@KRBTEST.COM for oryx/127.11.102.62@KRBTEST.COM
I20260620 06:07:14.310976 18331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:07:14.305772 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.102.60:41691 (local address 127.0.0.1:56110)
0620 06:07:14.306118 (+ 346us) negotiation.cc:107] Waiting for socket to connect
0620 06:07:14.306133 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 06:07:14.306215 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:07:14.306542 (+ 327us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:07:14.306548 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:07:14.306571 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 06:07:14.306682 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:07:14.306690 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:07:14.307908 (+ 1218us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:07:14.307914 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:07:14.309279 (+ 1365us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:07:14.309294 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:07:14.310480 (+ 1186us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:07:14.310486 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:07:14.310614 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:07:14.310651 (+ 37us) client_negotiation.cc:769] Sending connection context
0620 06:07:14.310706 (+ 55us) client_negotiation.cc:240] Negotiation successful
0620 06:07:14.310798 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":248}
I20260620 06:07:14.311204 18881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:07:14.294196 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.102.60:41691 (local address 127.11.102.2:60539)
0620 06:07:14.294429 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0620 06:07:14.294440 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 06:07:14.296581 (+ 2141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:07:14.297032 (+ 451us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:07:14.297043 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:07:14.297638 (+ 595us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:07:14.298474 (+ 836us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:07:14.298495 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:07:14.299732 (+ 1237us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:07:14.299735 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:07:14.301009 (+ 1274us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:07:14.301018 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:07:14.302004 (+ 986us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:07:14.302007 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:07:14.302100 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:07:14.302644 (+ 544us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:07:14.302668 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:07:14.305348 (+ 2680us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:07:14.307794 (+ 2446us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:07:14.307797 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:07:14.307799 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:07:14.308183 (+ 384us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:07:14.308516 (+ 333us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:07:14.308521 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:07:14.308525 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:07:14.308690 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:07:14.309212 (+ 522us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:07:14.309218 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:07:14.309533 (+ 315us) client_negotiation.cc:769] Sending connection context
0620 06:07:14.309750 (+ 217us) client_negotiation.cc:240] Negotiation successful
0620 06:07:14.310120 (+ 370us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":164,"mutex_wait_us":186,"spinlock_wait_cycles":22784,"thread_start_us":73,"threads_started":1}
I20260620 06:07:14.311852 18745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:07:14.294322 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.102.2:60539 (local address 127.11.102.60:41691)
0620 06:07:14.294494 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0620 06:07:14.294499 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 06:07:14.294516 (+ 17us) server_negotiation.cc:407] Connection header received
0620 06:07:14.296702 (+ 2186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:07:14.296708 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:07:14.296771 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:07:14.296829 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:07:14.298642 (+ 1813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:07:14.299561 (+ 919us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:07:14.301470 (+ 1909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:07:14.301784 (+ 314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:07:14.301880 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:07:14.305523 (+ 3643us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:07:14.305530 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:07:14.305536 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:07:14.305570 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:07:14.307695 (+ 2125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:07:14.308302 (+ 607us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:07:14.308305 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:07:14.308307 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:07:14.308391 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:07:14.308850 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:07:14.308853 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:07:14.308855 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:07:14.309065 (+ 210us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:07:14.309162 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0620 06:07:14.311473 (+ 2311us) server_negotiation.cc:299] Negotiation successful
0620 06:07:14.311599 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260620 06:07:14.312163 18883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:07:14.305864 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56110 (local address 127.11.102.60:41691)
0620 06:07:14.306141 (+ 277us) server_negotiation.cc:206] Beginning negotiation
0620 06:07:14.306147 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 06:07:14.306165 (+ 18us) server_negotiation.cc:407] Connection header received
0620 06:07:14.306320 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:07:14.306325 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:07:14.306393 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:07:14.306782 (+ 389us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 06:07:14.306892 (+ 110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:07:14.307764 (+ 872us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:07:14.309486 (+ 1722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:07:14.310318 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:07:14.311273 (+ 955us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:07:14.311461 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0620 06:07:14.311505 (+ 44us) server_negotiation.cc:299] Negotiation successful
0620 06:07:14.311593 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":189,"thread_start_us":95,"threads_started":1}
I20260620 06:07:14.312820 18569 raft_consensus.cc:1203] T 00000000000000000000000000000000 P ec929d712e20476684cecae76f307d8e [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20260620 06:07:14.313261 18612 sys_catalog.cc:455] T 00000000000000000000000000000000 P ec929d712e20476684cecae76f307d8e [sys.catalog]: SysCatalogTable state changed. Reason: New leader d7d4500dc38a48a1858293d72d1fcfc2. Latest consensus state: current_term: 1 leader_uuid: "d7d4500dc38a48a1858293d72d1fcfc2" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "d7d4500dc38a48a1858293d72d1fcfc2" member_type: VOTER last_known_addr { host: "127.11.102.62" port: 44777 } } peers { permanent_uuid: "0a9e7156fd5c4d399486dc5932b407b2" member_type: VOTER last_known_addr { host: "127.11.102.61" port: 39977 } attrs { promote: false } } peers { permanent_uuid: "ec929d712e20476684cecae76f307d8e" member_type: NON_VOTER last_known_addr { host: "127.11.102.60" port: 41691 } attrs { promote: true } } }
I20260620 06:07:14.313391 18612 sys_catalog.cc:458] T 00000000000000000000000000000000 P ec929d712e20476684cecae76f307d8e [sys.catalog]: This master's current role is: LEARNER
I20260620 06:07:14.313436 18498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:07:14.292871 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.102.2:57753 (local address 127.11.102.61:39977)
0620 06:07:14.293046 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0620 06:07:14.293051 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 06:07:14.293837 (+ 786us) server_negotiation.cc:407] Connection header received
0620 06:07:14.296545 (+ 2708us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:07:14.296552 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:07:14.296615 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:07:14.296671 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:07:14.298651 (+ 1980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:07:14.299343 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:07:14.301128 (+ 1785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:07:14.301476 (+ 348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:07:14.301570 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:07:14.305657 (+ 4087us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:07:14.305663 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:07:14.305670 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:07:14.305705 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:07:14.307580 (+ 1875us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:07:14.311915 (+ 4335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:07:14.311919 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:07:14.311921 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:07:14.312020 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:07:14.312500 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:07:14.312506 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:07:14.312511 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:07:14.312777 (+ 266us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:07:14.312894 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0620 06:07:14.313166 (+ 272us) server_negotiation.cc:299] Negotiation successful
0620 06:07:14.313285 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260620 06:07:14.313822 18879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:07:14.293243 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.102.61:39977 (local address 127.11.102.2:57753)
0620 06:07:14.293695 (+ 452us) negotiation.cc:107] Waiting for socket to connect
0620 06:07:14.293744 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0620 06:07:14.296304 (+ 2560us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:07:14.297032 (+ 728us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:07:14.297043 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:07:14.297632 (+ 589us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:07:14.298474 (+ 842us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:07:14.298495 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:07:14.299490 (+ 995us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:07:14.299492 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:07:14.301008 (+ 1516us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:07:14.301018 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:07:14.301723 (+ 705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:07:14.301727 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:07:14.301913 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:07:14.302778 (+ 865us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:07:14.302797 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:07:14.305497 (+ 2700us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:07:14.307759 (+ 2262us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:07:14.307765 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:07:14.307782 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:07:14.311800 (+ 4018us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:07:14.312285 (+ 485us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:07:14.312288 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:07:14.312289 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:07:14.312375 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:07:14.312901 (+ 526us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:07:14.312903 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:07:14.312995 (+ 92us) client_negotiation.cc:769] Sending connection context
0620 06:07:14.313096 (+ 101us) client_negotiation.cc:240] Negotiation successful
0620 06:07:14.313221 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":265,"mutex_wait_us":155,"spinlock_wait_cycles":11776,"thread_start_us":131,"threads_started":1}
I20260620 06:07:14.314273 18875 heartbeater.cc:347] Connected to a master server at 127.11.102.60:41691
I20260620 06:07:14.314529 18875 heartbeater.cc:464] Registering TS with master...
I20260620 06:07:14.314543 18877 heartbeater.cc:347] Connected to a master server at 127.11.102.61:39977
I20260620 06:07:14.314647 18877 heartbeater.cc:464] Registering TS with master...
I20260620 06:07:14.315135 18875 heartbeater.cc:511] Master 127.11.102.60:41691 requested a full tablet report, sending...
I20260620 06:07:14.315135 18877 heartbeater.cc:511] Master 127.11.102.61:39977 requested a full tablet report, sending...
I20260620 06:07:14.315882 17813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:07:14.292837 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.102.2:40299 (local address 127.11.102.62:44777)
0620 06:07:14.292970 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0620 06:07:14.292974 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 06:07:14.293841 (+ 867us) server_negotiation.cc:407] Connection header received
0620 06:07:14.301160 (+ 7319us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:07:14.301163 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:07:14.301213 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:07:14.301296 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:07:14.301797 (+ 501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:07:14.302594 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:07:14.303643 (+ 1049us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:07:14.304009 (+ 366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:07:14.304118 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:07:14.309231 (+ 5113us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:07:14.309234 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:07:14.309237 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:07:14.309262 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:07:14.310963 (+ 1701us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:07:14.314553 (+ 3590us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:07:14.314557 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:07:14.314559 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:07:14.314648 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:07:14.314985 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:07:14.314988 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:07:14.314989 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:07:14.315215 (+ 226us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:07:14.315300 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0620 06:07:14.315621 (+ 321us) server_negotiation.cc:299] Negotiation successful
0620 06:07:14.315750 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260620 06:07:14.316308 18367 ts_manager.cc:194] Registered new tserver with Master: f569ba0a0ea74d078dabe54eb42b9e21 (127.11.102.2:33643)
I20260620 06:07:14.316367 18567 ts_manager.cc:194] Registered new tserver with Master: f569ba0a0ea74d078dabe54eb42b9e21 (127.11.102.2:33643)
I20260620 06:07:14.316795 18880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:07:14.293243 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.102.62:44777 (local address 127.11.102.2:40299)
0620 06:07:14.293695 (+ 452us) negotiation.cc:107] Waiting for socket to connect
0620 06:07:14.293744 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0620 06:07:14.296297 (+ 2553us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:07:14.301316 (+ 5019us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:07:14.301318 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:07:14.301532 (+ 214us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:07:14.301644 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:07:14.301651 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:07:14.302739 (+ 1088us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:07:14.302745 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:07:14.303509 (+ 764us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:07:14.303518 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:07:14.304294 (+ 776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:07:14.304300 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:07:14.304449 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:07:14.305000 (+ 551us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:07:14.305015 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:07:14.309090 (+ 4075us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:07:14.311092 (+ 2002us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:07:14.311095 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:07:14.311097 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:07:14.314441 (+ 3344us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:07:14.314779 (+ 338us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:07:14.314781 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:07:14.314783 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:07:14.314862 (+ 79us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:07:14.315354 (+ 492us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:07:14.315357 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:07:14.315431 (+ 74us) client_negotiation.cc:769] Sending connection context
0620 06:07:14.316508 (+ 1077us) client_negotiation.cc:240] Negotiation successful
0620 06:07:14.316642 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":285,"mutex_wait_us":21,"spinlock_wait_cycles":23296,"thread_start_us":139,"threads_started":1}
I20260620 06:07:14.317329 18876 heartbeater.cc:347] Connected to a master server at 127.11.102.62:44777
I20260620 06:07:14.317426 18876 heartbeater.cc:464] Registering TS with master...
I20260620 06:07:14.317580 18876 heartbeater.cc:511] Master 127.11.102.62:44777 requested a full tablet report, sending...
I20260620 06:07:14.318212 17774 ts_manager.cc:194] Re-registered known tserver with Master: f569ba0a0ea74d078dabe54eb42b9e21 (127.11.102.2:33643)
I20260620 06:07:14.318907 17774 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.11.102.2@KRBTEST.COM'} at 127.11.102.2:40299
W20260620 06:07:14.415524 18882 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:07:14.415764 18882 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:07:14.415804 18882 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:07:14.419644 18882 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:07:14.419689 18882 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:07:14.419744 18882 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.102.3
I20260620 06:07:14.424595 18882 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.102.20:34945
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935603120653-11672-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935603120653-11672-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935603120653-11672-0/krb5kdc/oryx_127.11.102.3.keytab
--principal=oryx/127.11.102.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.102.3:36875
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935603120653-11672-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.11.102.3
--webserver_port=43289
--tserver_master_addrs=127.11.102.62:44777,127.11.102.61:39977,127.11.102.60:41691
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18882
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.102.3
--log_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935603120653-11672-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 05:28:48 UTC on ccycloud.quasar-fixhms.root.comops.site
build id 4457020
I20260620 06:07:14.425463 18882 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:07:14.426398 18882 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 06:07:14.435954 18891 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:07:14.436057 18888 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:07:14.436053 18889 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:07:14.436455 18882 server_base.cc:1034] running on GCE node
I20260620 06:07:14.436816 18882 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:07:14.437453 18882 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:07:14.438637 18882 hybrid_clock.cc:648] HybridClock initialized: now 1781935634438616 us; error 32 us; skew 500 ppm
Jun 20 06:07:14 dist-test-slave-9hm5 krb5kdc[17729](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935634, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.102.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 06:07:14.441421 18882 init.cc:377] Logged in from keytab as oryx/127.11.102.3@KRBTEST.COM (short username oryx)
I20260620 06:07:14.442217 18882 webserver.cc:460] Webserver started at http://127.11.102.3:43289/ using document root <none> and password file <none>
I20260620 06:07:14.442736 18882 fs_manager.cc:359] Metadata directory not provided
I20260620 06:07:14.442831 18882 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:07:14.446889 18882 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260620 06:07:14.448937 18898 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:07:14.450096 18882 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260620 06:07:14.450237 18882 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935603120653-11672-0/minicluster-data/ts-2/data,/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935603120653-11672-0/minicluster-data/ts-2/wal
uuid: "d4b3fbe7b8664c968f75cc26b72410d7"
format_stamp: "Formatted at 2026-06-20 06:07:13 on dist-test-slave-9hm5"
server_key: "fd44a5564a95678a66d44c25afe1a9e8"
server_key_iv: "1faf2a938ebbaba58ec523f52a54b31f"
server_key_version: "encryptionkey@0"
I20260620 06:07:14.450800 18882 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935603120653-11672-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935603120653-11672-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935603120653-11672-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:07:14.463348 18882 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:07:14.466737 18882 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:07:14.466970 18882 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:07:14.467561 18882 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 06:07:14.468493 18882 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 06:07:14.468567 18882 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 06:07:14.468640 18882 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 06:07:14.468684 18882 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 06:07:14.478308 18882 rpc_server.cc:307] RPC server started. Bound to: 127.11.102.3:36875
I20260620 06:07:14.478327 19011 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.102.3:36875 every 8 connection(s)
I20260620 06:07:14.479420 18882 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935603120653-11672-0/minicluster-data/ts-2/data/info.pb
I20260620 06:07:14.480003 11672 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9HMp43/build/debug/bin/kudu as pid 18882
Jun 20 06:07:14 dist-test-slave-9hm5 krb5kdc[17729](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935634, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.102.3@KRBTEST.COM for oryx/127.11.102.60@KRBTEST.COM
I20260620 06:07:14.493610 18883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:07:14.482329 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56114 (local address 127.11.102.60:41691)
0620 06:07:14.482466 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0620 06:07:14.482470 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 06:07:14.483222 (+ 752us) server_negotiation.cc:407] Connection header received
0620 06:07:14.483415 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:07:14.483418 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:07:14.483481 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:07:14.483537 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:07:14.484616 (+ 1079us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:07:14.485429 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:07:14.486671 (+ 1242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:07:14.487003 (+ 332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:07:14.487069 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:07:14.489418 (+ 2349us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:07:14.489421 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:07:14.489424 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:07:14.489461 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:07:14.491212 (+ 1751us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:07:14.491826 (+ 614us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:07:14.491833 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:07:14.491836 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:07:14.491919 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:07:14.492234 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:07:14.492237 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:07:14.492239 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:07:14.492454 (+ 215us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:07:14.492575 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0620 06:07:14.493271 (+ 696us) server_negotiation.cc:299] Negotiation successful
0620 06:07:14.493415 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260620 06:07:14.494939 19022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:07:14.482632 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54674 (local address 127.11.102.62:44777)
0620 06:07:14.484172 (+ 1540us) server_negotiation.cc:206] Beginning negotiation
0620 06:07:14.484178 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 06:07:14.484194 (+ 16us) server_negotiation.cc:407] Connection header received
0620 06:07:14.484240 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:07:14.484243 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:07:14.484294 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:07:14.484393 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:07:14.485430 (+ 1037us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:07:14.486376 (+ 946us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:07:14.487671 (+ 1295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:07:14.488034 (+ 363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:07:14.488512 (+ 478us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:07:14.489812 (+ 1300us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:07:14.489816 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:07:14.489819 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:07:14.489857 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:07:14.492227 (+ 2370us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:07:14.493174 (+ 947us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:07:14.493180 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:07:14.493183 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:07:14.493274 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:07:14.493586 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:07:14.493589 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:07:14.493591 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:07:14.493897 (+ 306us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:07:14.494007 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0620 06:07:14.494580 (+ 573us) server_negotiation.cc:299] Negotiation successful
0620 06:07:14.494717 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1448,"thread_start_us":66,"threads_started":1}
Jun 20 06:07:14 dist-test-slave-9hm5 krb5kdc[17729](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935634, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.102.3@KRBTEST.COM for oryx/127.11.102.62@KRBTEST.COM
Jun 20 06:07:14 dist-test-slave-9hm5 krb5kdc[17729](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935634, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.102.3@KRBTEST.COM for oryx/127.11.102.61@KRBTEST.COM
I20260620 06:07:14.499282 18745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:07:14.483440 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.102.3:39739 (local address 127.11.102.60:41691)
0620 06:07:14.483556 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0620 06:07:14.483561 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 06:07:14.484775 (+ 1214us) server_negotiation.cc:407] Connection header received
0620 06:07:14.486023 (+ 1248us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:07:14.486026 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:07:14.486067 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:07:14.486113 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:07:14.486922 (+ 809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:07:14.487602 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:07:14.489403 (+ 1801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:07:14.489751 (+ 348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:07:14.489841 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:07:14.494650 (+ 4809us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:07:14.494653 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:07:14.494656 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:07:14.494678 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:07:14.496819 (+ 2141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:07:14.497364 (+ 545us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:07:14.497368 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:07:14.497370 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:07:14.497455 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:07:14.497895 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:07:14.497899 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:07:14.497903 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:07:14.498161 (+ 258us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:07:14.498236 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0620 06:07:14.498991 (+ 755us) server_negotiation.cc:299] Negotiation successful
0620 06:07:14.499110 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260620 06:07:14.501048 19021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:07:14.483094 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.102.60:41691 (local address 127.11.102.3:39739)
0620 06:07:14.484702 (+ 1608us) negotiation.cc:107] Waiting for socket to connect
0620 06:07:14.484716 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 06:07:14.485891 (+ 1175us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:07:14.486229 (+ 338us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:07:14.486231 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:07:14.486393 (+ 162us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:07:14.486815 (+ 422us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:07:14.486823 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:07:14.487796 (+ 973us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:07:14.487802 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:07:14.489284 (+ 1482us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:07:14.489293 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:07:14.490069 (+ 776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:07:14.490074 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:07:14.490312 (+ 238us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:07:14.491027 (+ 715us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:07:14.491051 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:07:14.494481 (+ 3430us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:07:14.496930 (+ 2449us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:07:14.496935 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:07:14.496947 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:07:14.497237 (+ 290us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:07:14.497580 (+ 343us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:07:14.497584 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:07:14.497586 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:07:14.497765 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:07:14.498329 (+ 564us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:07:14.498336 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:07:14.498670 (+ 334us) client_negotiation.cc:769] Sending connection context
0620 06:07:14.499434 (+ 764us) client_negotiation.cc:240] Negotiation successful
0620 06:07:14.499849 (+ 415us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1431,"mutex_wait_us":293,"thread_start_us":113,"threads_started":1}
I20260620 06:07:14.501322 17813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:07:14.481420 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.102.3:39027 (local address 127.11.102.62:44777)
0620 06:07:14.481554 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0620 06:07:14.481559 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 06:07:14.483646 (+ 2087us) server_negotiation.cc:407] Connection header received
0620 06:07:14.484990 (+ 1344us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:07:14.484993 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:07:14.485040 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:07:14.485087 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:07:14.487186 (+ 2099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:07:14.488034 (+ 848us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:07:14.492028 (+ 3994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:07:14.492446 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:07:14.492556 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:07:14.497864 (+ 5308us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:07:14.497869 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:07:14.497872 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:07:14.497898 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:07:14.499689 (+ 1791us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:07:14.500162 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:07:14.500165 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:07:14.500167 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:07:14.500242 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:07:14.500520 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:07:14.500523 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:07:14.500525 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:07:14.500698 (+ 173us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:07:14.500772 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0620 06:07:14.501086 (+ 314us) server_negotiation.cc:299] Negotiation successful
0620 06:07:14.501187 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260620 06:07:14.502089 19024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:07:14.481950 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.102.62:44777 (local address 127.11.102.3:39027)
0620 06:07:14.483477 (+ 1527us) negotiation.cc:107] Waiting for socket to connect
0620 06:07:14.483531 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0620 06:07:14.484798 (+ 1267us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:07:14.485448 (+ 650us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:07:14.485458 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:07:14.486943 (+ 1485us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:07:14.487058 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:07:14.487064 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:07:14.488175 (+ 1111us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:07:14.488178 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:07:14.490461 (+ 2283us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:07:14.490472 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:07:14.492681 (+ 2209us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:07:14.492685 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:07:14.492795 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:07:14.493277 (+ 482us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:07:14.493293 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:07:14.497708 (+ 4415us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:07:14.499802 (+ 2094us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:07:14.499805 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:07:14.499807 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:07:14.500032 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:07:14.500340 (+ 308us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:07:14.500343 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:07:14.500345 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:07:14.500418 (+ 73us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:07:14.500824 (+ 406us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:07:14.500827 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:07:14.500906 (+ 79us) client_negotiation.cc:769] Sending connection context
0620 06:07:14.501508 (+ 602us) client_negotiation.cc:240] Negotiation successful
0620 06:07:14.501634 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1347,"mutex_wait_us":1363,"spinlock_wait_cycles":33152,"thread_start_us":160,"threads_started":1}
I20260620 06:07:14.502283 19012 heartbeater.cc:347] Connected to a master server at 127.11.102.60:41691
I20260620 06:07:14.502512 19012 heartbeater.cc:464] Registering TS with master...
I20260620 06:07:14.502902 19013 heartbeater.cc:347] Connected to a master server at 127.11.102.62:44777
I20260620 06:07:14.502995 19013 heartbeater.cc:464] Registering TS with master...
I20260620 06:07:14.503031 19012 heartbeater.cc:511] Master 127.11.102.60:41691 requested a full tablet report, sending...
I20260620 06:07:14.499873 17774 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54674:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260620 06:07:14.503180 19013 heartbeater.cc:511] Master 127.11.102.62:44777 requested a full tablet report, sending...
I20260620 06:07:14.504400 18567 ts_manager.cc:194] Registered new tserver with Master: d4b3fbe7b8664c968f75cc26b72410d7 (127.11.102.3:36875)
I20260620 06:07:14.504413 17773 ts_manager.cc:194] Re-registered known tserver with Master: d4b3fbe7b8664c968f75cc26b72410d7 (127.11.102.3:36875)
I20260620 06:07:14.505122 17773 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.11.102.3@KRBTEST.COM'} at 127.11.102.3:39027
I20260620 06:07:14.505671 19027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:07:14.481950 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.102.61:39977 (local address 127.11.102.3:42101)
0620 06:07:14.483529 (+ 1579us) negotiation.cc:107] Waiting for socket to connect
0620 06:07:14.483546 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 06:07:14.484744 (+ 1198us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:07:14.485449 (+ 705us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:07:14.485458 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:07:14.486016 (+ 558us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:07:14.486738 (+ 722us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:07:14.486757 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:07:14.490188 (+ 3431us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:07:14.490193 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:07:14.491231 (+ 1038us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:07:14.491242 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:07:14.494651 (+ 3409us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:07:14.494658 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:07:14.494760 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:07:14.495856 (+ 1096us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:07:14.495874 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:07:14.497580 (+ 1706us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:07:14.502233 (+ 4653us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:07:14.502236 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:07:14.502238 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:07:14.503773 (+ 1535us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:07:14.504127 (+ 354us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:07:14.504129 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:07:14.504131 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:07:14.504212 (+ 81us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:07:14.505113 (+ 901us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:07:14.505115 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:07:14.505225 (+ 110us) client_negotiation.cc:769] Sending connection context
0620 06:07:14.505361 (+ 136us) client_negotiation.cc:240] Negotiation successful
0620 06:07:14.505507 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1303,"mutex_wait_us":1,"spinlock_wait_cycles":7424,"thread_start_us":692,"threads_started":1}
I20260620 06:07:14.505712 18498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:07:14.481846 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.102.3:42101 (local address 127.11.102.61:39977)
0620 06:07:14.481973 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0620 06:07:14.481977 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 06:07:14.483658 (+ 1681us) server_negotiation.cc:407] Connection header received
0620 06:07:14.485017 (+ 1359us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:07:14.485022 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:07:14.485079 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:07:14.485138 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:07:14.489108 (+ 3970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:07:14.490023 (+ 915us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:07:14.491401 (+ 1378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:07:14.491816 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:07:14.491904 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:07:14.497727 (+ 5823us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:07:14.497731 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:07:14.497734 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:07:14.497759 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:07:14.499941 (+ 2182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:07:14.503906 (+ 3965us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:07:14.503910 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:07:14.503912 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:07:14.503996 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:07:14.504311 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:07:14.504315 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:07:14.504316 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:07:14.504565 (+ 249us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:07:14.504695 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0620 06:07:14.505427 (+ 732us) server_negotiation.cc:299] Negotiation successful
0620 06:07:14.505538 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
W20260620 06:07:14.505703 17774 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 06:07:14.506512 19023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:07:14.482653 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52498 (local address 127.11.102.61:39977)
0620 06:07:14.483231 (+ 578us) server_negotiation.cc:206] Beginning negotiation
0620 06:07:14.483236 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 06:07:14.486724 (+ 3488us) server_negotiation.cc:407] Connection header received
0620 06:07:14.490731 (+ 4007us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:07:14.490736 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:07:14.490787 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:07:14.490847 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:07:14.493347 (+ 2500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:07:14.494088 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:07:14.495158 (+ 1070us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:07:14.495480 (+ 322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:07:14.495589 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:07:14.498832 (+ 3243us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:07:14.498836 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:07:14.498839 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:07:14.498873 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:07:14.503774 (+ 4901us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:07:14.504887 (+ 1113us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:07:14.504891 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:07:14.504893 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:07:14.504977 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:07:14.505325 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:07:14.505329 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:07:14.505331 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:07:14.505537 (+ 206us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:07:14.505660 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0620 06:07:14.506233 (+ 573us) server_negotiation.cc:299] Negotiation successful
0620 06:07:14.506350 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":502,"thread_start_us":77,"threads_started":1}
I20260620 06:07:14.506531 19014 heartbeater.cc:347] Connected to a master server at 127.11.102.61:39977
I20260620 06:07:14.506599 19014 heartbeater.cc:464] Registering TS with master...
I20260620 06:07:14.506737 19014 heartbeater.cc:511] Master 127.11.102.61:39977 requested a full tablet report, sending...
I20260620 06:07:14.507807 18367 ts_manager.cc:194] Registered new tserver with Master: d4b3fbe7b8664c968f75cc26b72410d7 (127.11.102.3:36875)
I20260620 06:07:14.508867 18335 sys_catalog.cc:455] T 00000000000000000000000000000000 P d7d4500dc38a48a1858293d72d1fcfc2 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "d7d4500dc38a48a1858293d72d1fcfc2" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "d7d4500dc38a48a1858293d72d1fcfc2" member_type: VOTER last_known_addr { host: "127.11.102.62" port: 44777 } } peers { permanent_uuid: "0a9e7156fd5c4d399486dc5932b407b2" member_type: VOTER last_known_addr { host: "127.11.102.61" port: 39977 } attrs { promote: false } } peers { permanent_uuid: "ec929d712e20476684cecae76f307d8e" member_type: NON_VOTER last_known_addr { host: "127.11.102.60" port: 41691 } attrs { promote: true } } }
I20260620 06:07:14.509006 18335 sys_catalog.cc:458] T 00000000000000000000000000000000 P d7d4500dc38a48a1858293d72d1fcfc2 [sys.catalog]: This master's current role is: LEADER
I20260620 06:07:14.512125 19031 raft_consensus.cc:1050] T 00000000000000000000000000000000 P d7d4500dc38a48a1858293d72d1fcfc2: attempting to promote NON_VOTER ec929d712e20476684cecae76f307d8e to VOTER
I20260620 06:07:14.512449 19031 consensus_queue.cc:237] T 00000000000000000000000000000000 P d7d4500dc38a48a1858293d72d1fcfc2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "d7d4500dc38a48a1858293d72d1fcfc2" member_type: VOTER last_known_addr { host: "127.11.102.62" port: 44777 } } peers { permanent_uuid: "0a9e7156fd5c4d399486dc5932b407b2" member_type: VOTER last_known_addr { host: "127.11.102.61" port: 39977 } attrs { promote: false } } peers { permanent_uuid: "ec929d712e20476684cecae76f307d8e" member_type: VOTER last_known_addr { host: "127.11.102.60" port: 41691 } attrs { promote: false } }
I20260620 06:07:14.513541 18381 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 0a9e7156fd5c4d399486dc5932b407b2 [term 1 FOLLOWER]: Refusing update from remote peer d7d4500dc38a48a1858293d72d1fcfc2: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260620 06:07:14.513893 19029 consensus_queue.cc:1035] T 00000000000000000000000000000000 P d7d4500dc38a48a1858293d72d1fcfc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a9e7156fd5c4d399486dc5932b407b2" member_type: VOTER last_known_addr { host: "127.11.102.61" port: 39977 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260620 06:07:14.515228 18569 raft_consensus.cc:1261] T 00000000000000000000000000000000 P ec929d712e20476684cecae76f307d8e [term 1 LEARNER]: Refusing update from remote peer d7d4500dc38a48a1858293d72d1fcfc2: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260620 06:07:14.515569 18335 consensus_queue.cc:1035] T 00000000000000000000000000000000 P d7d4500dc38a48a1858293d72d1fcfc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec929d712e20476684cecae76f307d8e" member_type: VOTER last_known_addr { host: "127.11.102.60" port: 41691 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260620 06:07:14.515995 18335 raft_consensus.cc:2932] T 00000000000000000000000000000000 P d7d4500dc38a48a1858293d72d1fcfc2 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 7 to 9, ec929d712e20476684cecae76f307d8e (127.11.102.60) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "d7d4500dc38a48a1858293d72d1fcfc2" member_type: VOTER last_known_addr { host: "127.11.102.62" port: 44777 } } peers { permanent_uuid: "0a9e7156fd5c4d399486dc5932b407b2" member_type: VOTER last_known_addr { host: "127.11.102.61" port: 39977 } attrs { promote: false } } peers { permanent_uuid: "ec929d712e20476684cecae76f307d8e" member_type: VOTER last_known_addr { host: "127.11.102.60" port: 41691 } attrs { promote: false } } }
I20260620 06:07:14.516495 18381 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 0a9e7156fd5c4d399486dc5932b407b2 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, ec929d712e20476684cecae76f307d8e (127.11.102.60) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "d7d4500dc38a48a1858293d72d1fcfc2" member_type: VOTER last_known_addr { host: "127.11.102.62" port: 44777 } } peers { permanent_uuid: "0a9e7156fd5c4d399486dc5932b407b2" member_type: VOTER last_known_addr { host: "127.11.102.61" port: 39977 } attrs { promote: false } } peers { permanent_uuid: "ec929d712e20476684cecae76f307d8e" member_type: VOTER last_known_addr { host: "127.11.102.60" port: 41691 } attrs { promote: false } } }
I20260620 06:07:14.517719 19029 sys_catalog.cc:455] T 00000000000000000000000000000000 P d7d4500dc38a48a1858293d72d1fcfc2 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "d7d4500dc38a48a1858293d72d1fcfc2" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "d7d4500dc38a48a1858293d72d1fcfc2" member_type: VOTER last_known_addr { host: "127.11.102.62" port: 44777 } } peers { permanent_uuid: "0a9e7156fd5c4d399486dc5932b407b2" member_type: VOTER last_known_addr { host: "127.11.102.61" port: 39977 } attrs { promote: false } } peers { permanent_uuid: "ec929d712e20476684cecae76f307d8e" member_type: VOTER last_known_addr { host: "127.11.102.60" port: 41691 } attrs { promote: false } } }
I20260620 06:07:14.517832 19029 sys_catalog.cc:458] T 00000000000000000000000000000000 P d7d4500dc38a48a1858293d72d1fcfc2 [sys.catalog]: This master's current role is: LEADER
I20260620 06:07:14.517972 18335 sys_catalog.cc:455] T 00000000000000000000000000000000 P d7d4500dc38a48a1858293d72d1fcfc2 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "d7d4500dc38a48a1858293d72d1fcfc2" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "d7d4500dc38a48a1858293d72d1fcfc2" member_type: VOTER last_known_addr { host: "127.11.102.62" port: 44777 } } peers { permanent_uuid: "0a9e7156fd5c4d399486dc5932b407b2" member_type: VOTER last_known_addr { host: "127.11.102.61" port: 39977 } attrs { promote: false } } peers { permanent_uuid: "ec929d712e20476684cecae76f307d8e" member_type: VOTER last_known_addr { host: "127.11.102.60" port: 41691 } attrs { promote: false } } }
I20260620 06:07:14.518062 18335 sys_catalog.cc:458] T 00000000000000000000000000000000 P d7d4500dc38a48a1858293d72d1fcfc2 [sys.catalog]: This master's current role is: LEADER
I20260620 06:07:14.518244 18335 sys_catalog.cc:455] T 00000000000000000000000000000000 P d7d4500dc38a48a1858293d72d1fcfc2 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "d7d4500dc38a48a1858293d72d1fcfc2" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "d7d4500dc38a48a1858293d72d1fcfc2" member_type: VOTER last_known_addr { host: "127.11.102.62" port: 44777 } } peers { permanent_uuid: "0a9e7156fd5c4d399486dc5932b407b2" member_type: VOTER last_known_addr { host: "127.11.102.61" port: 39977 } attrs { promote: false } } peers { permanent_uuid: "ec929d712e20476684cecae76f307d8e" member_type: VOTER last_known_addr { host: "127.11.102.60" port: 41691 } attrs { promote: false } } }
I20260620 06:07:14.518326 18335 sys_catalog.cc:458] T 00000000000000000000000000000000 P d7d4500dc38a48a1858293d72d1fcfc2 [sys.catalog]: This master's current role is: LEADER
I20260620 06:07:14.518076 18569 raft_consensus.cc:2932] T 00000000000000000000000000000000 P ec929d712e20476684cecae76f307d8e [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, ec929d712e20476684cecae76f307d8e (127.11.102.60) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "d7d4500dc38a48a1858293d72d1fcfc2" member_type: VOTER last_known_addr { host: "127.11.102.62" port: 44777 } } peers { permanent_uuid: "0a9e7156fd5c4d399486dc5932b407b2" member_type: VOTER last_known_addr { host: "127.11.102.61" port: 39977 } attrs { promote: false } } peers { permanent_uuid: "ec929d712e20476684cecae76f307d8e" member_type: VOTER last_known_addr { host: "127.11.102.60" port: 41691 } attrs { promote: false } } }
I20260620 06:07:14.522275 18612 sys_catalog.cc:455] T 00000000000000000000000000000000 P ec929d712e20476684cecae76f307d8e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "d7d4500dc38a48a1858293d72d1fcfc2" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "d7d4500dc38a48a1858293d72d1fcfc2" member_type: VOTER last_known_addr { host: "127.11.102.62" port: 44777 } } peers { permanent_uuid: "0a9e7156fd5c4d399486dc5932b407b2" member_type: VOTER last_known_addr { host: "127.11.102.61" port: 39977 } attrs { promote: false } } peers { permanent_uuid: "ec929d712e20476684cecae76f307d8e" member_type: VOTER last_known_addr { host: "127.11.102.60" port: 41691 } attrs { promote: false } } }
I20260620 06:07:14.522404 18612 sys_catalog.cc:458] T 00000000000000000000000000000000 P ec929d712e20476684cecae76f307d8e [sys.catalog]: This master's current role is: FOLLOWER
I20260620 06:07:14.523033 19039 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0a9e7156fd5c4d399486dc5932b407b2 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "d7d4500dc38a48a1858293d72d1fcfc2" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "d7d4500dc38a48a1858293d72d1fcfc2" member_type: VOTER last_known_addr { host: "127.11.102.62" port: 44777 } } peers { permanent_uuid: "0a9e7156fd5c4d399486dc5932b407b2" member_type: VOTER last_known_addr { host: "127.11.102.61" port: 39977 } attrs { promote: false } } peers { permanent_uuid: "ec929d712e20476684cecae76f307d8e" member_type: VOTER last_known_addr { host: "127.11.102.60" port: 41691 } attrs { promote: false } } }
I20260620 06:07:14.523129 19039 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0a9e7156fd5c4d399486dc5932b407b2 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 06:07:14.528101 19041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:07:14.520285 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.102.1:37681 (local address 127.0.0.1:52236)
0620 06:07:14.520520 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0620 06:07:14.520534 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 06:07:14.520624 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:07:14.524269 (+ 3645us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:07:14.524272 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:07:14.524289 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 06:07:14.524402 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:07:14.524409 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:07:14.525815 (+ 1406us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:07:14.525818 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:07:14.526735 (+ 917us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:07:14.526743 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:07:14.527653 (+ 910us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:07:14.527656 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:07:14.527742 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:07:14.527767 (+ 25us) client_negotiation.cc:769] Sending connection context
0620 06:07:14.527867 (+ 100us) client_negotiation.cc:240] Negotiation successful
0620 06:07:14.527948 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":171,"mutex_wait_us":87,"thread_start_us":112,"threads_started":1}
I20260620 06:07:14.528133 19044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:07:14.520690 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52236 (local address 127.11.102.1:37681)
0620 06:07:14.523877 (+ 3187us) server_negotiation.cc:206] Beginning negotiation
0620 06:07:14.523886 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 06:07:14.523908 (+ 22us) server_negotiation.cc:407] Connection header received
0620 06:07:14.523971 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:07:14.523975 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:07:14.524133 (+ 158us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:07:14.524291 (+ 158us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 06:07:14.524517 (+ 226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:07:14.525687 (+ 1170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:07:14.526856 (+ 1169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:07:14.527508 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:07:14.527637 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:07:14.527776 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0620 06:07:14.527903 (+ 127us) server_negotiation.cc:299] Negotiation successful
0620 06:07:14.527989 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3064,"thread_start_us":147,"threads_started":1}
I20260620 06:07:14.528743 19042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:07:14.520297 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.102.3:36875 (local address 127.0.0.1:58686)
0620 06:07:14.523277 (+ 2980us) negotiation.cc:107] Waiting for socket to connect
0620 06:07:14.523288 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 06:07:14.523819 (+ 531us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:07:14.524459 (+ 640us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:07:14.524462 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:07:14.524475 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 06:07:14.524566 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:07:14.524571 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:07:14.525940 (+ 1369us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:07:14.525943 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:07:14.527167 (+ 1224us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:07:14.527175 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:07:14.528305 (+ 1130us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:07:14.528311 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:07:14.528407 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:07:14.528444 (+ 37us) client_negotiation.cc:769] Sending connection context
0620 06:07:14.528516 (+ 72us) client_negotiation.cc:240] Negotiation successful
0620 06:07:14.528582 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2931,"mutex_wait_us":63,"thread_start_us":97,"threads_started":1}
I20260620 06:07:14.528764 18331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:07:14.519675 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.102.2:33643 (local address 127.0.0.1:59304)
0620 06:07:14.519820 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0620 06:07:14.519835 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 06:07:14.519906 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:07:14.524345 (+ 4439us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:07:14.524348 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:07:14.524365 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 06:07:14.524459 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:07:14.524465 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:07:14.525796 (+ 1331us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:07:14.525802 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:07:14.527141 (+ 1339us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:07:14.527151 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:07:14.528383 (+ 1232us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:07:14.528389 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:07:14.528475 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:07:14.528520 (+ 45us) client_negotiation.cc:769] Sending connection context
0620 06:07:14.528573 (+ 53us) client_negotiation.cc:240] Negotiation successful
0620 06:07:14.528631 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":77,"mutex_wait_us":71}
I20260620 06:07:14.528776 19043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:07:14.520649 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58686 (local address 127.11.102.3:36875)
0620 06:07:14.520969 (+ 320us) server_negotiation.cc:206] Beginning negotiation
0620 06:07:14.520981 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 06:07:14.523928 (+ 2947us) server_negotiation.cc:407] Connection header received
0620 06:07:14.524005 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:07:14.524010 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:07:14.524196 (+ 186us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:07:14.524338 (+ 142us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 06:07:14.524789 (+ 451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:07:14.525838 (+ 1049us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:07:14.527283 (+ 1445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:07:14.528017 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:07:14.528111 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:07:14.528246 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0620 06:07:14.528560 (+ 314us) server_negotiation.cc:299] Negotiation successful
0620 06:07:14.528645 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":226,"thread_start_us":107,"threads_started":1}
I20260620 06:07:14.529521 19040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:07:14.520077 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59304 (local address 127.11.102.2:33643)
0620 06:07:14.523973 (+ 3896us) server_negotiation.cc:206] Beginning negotiation
0620 06:07:14.523984 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 06:07:14.524003 (+ 19us) server_negotiation.cc:407] Connection header received
0620 06:07:14.524063 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:07:14.524067 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:07:14.524204 (+ 137us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:07:14.524604 (+ 400us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 06:07:14.524769 (+ 165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:07:14.525685 (+ 916us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:07:14.527268 (+ 1583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:07:14.528090 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:07:14.529219 (+ 1129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:07:14.529314 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0620 06:07:14.529350 (+ 36us) server_negotiation.cc:299] Negotiation successful
0620 06:07:14.529422 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3786,"thread_start_us":107,"threads_started":1}
I20260620 06:07:14.531776 18946 tablet_service.cc:1467] Processing CreateTablet for tablet 0c7e523cec3b44c98e0b0cedea19ae30 (DEFAULT_TABLE table=test-table [id=6ba82862694d41929b13c412aa10527c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 06:07:14.531776 18672 tablet_service.cc:1467] Processing CreateTablet for tablet 0c7e523cec3b44c98e0b0cedea19ae30 (DEFAULT_TABLE table=test-table [id=6ba82862694d41929b13c412aa10527c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 06:07:14.531762 18809 tablet_service.cc:1467] Processing CreateTablet for tablet 0c7e523cec3b44c98e0b0cedea19ae30 (DEFAULT_TABLE table=test-table [id=6ba82862694d41929b13c412aa10527c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 06:07:14.532778 18809 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0c7e523cec3b44c98e0b0cedea19ae30. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 06:07:14.532779 18672 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0c7e523cec3b44c98e0b0cedea19ae30. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 06:07:14.532789 18946 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0c7e523cec3b44c98e0b0cedea19ae30. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 06:07:14.539219 19045 tablet_bootstrap.cc:492] T 0c7e523cec3b44c98e0b0cedea19ae30 P f569ba0a0ea74d078dabe54eb42b9e21: Bootstrap starting.
I20260620 06:07:14.540759 19046 tablet_bootstrap.cc:492] T 0c7e523cec3b44c98e0b0cedea19ae30 P d4b3fbe7b8664c968f75cc26b72410d7: Bootstrap starting.
I20260620 06:07:14.541569 19047 tablet_bootstrap.cc:492] T 0c7e523cec3b44c98e0b0cedea19ae30 P fb0783328ad3442fba39f260adf25308: Bootstrap starting.
I20260620 06:07:14.541692 19045 tablet_bootstrap.cc:654] T 0c7e523cec3b44c98e0b0cedea19ae30 P f569ba0a0ea74d078dabe54eb42b9e21: Neither blocks nor log segments found. Creating new log.
I20260620 06:07:14.542564 19045 log.cc:824] T 0c7e523cec3b44c98e0b0cedea19ae30 P f569ba0a0ea74d078dabe54eb42b9e21: Log is configured to *not* fsync() on all Append() calls
I20260620 06:07:14.542826 19046 tablet_bootstrap.cc:654] T 0c7e523cec3b44c98e0b0cedea19ae30 P d4b3fbe7b8664c968f75cc26b72410d7: Neither blocks nor log segments found. Creating new log.
I20260620 06:07:14.543538 19046 log.cc:824] T 0c7e523cec3b44c98e0b0cedea19ae30 P d4b3fbe7b8664c968f75cc26b72410d7: Log is configured to *not* fsync() on all Append() calls
I20260620 06:07:14.543598 19047 tablet_bootstrap.cc:654] T 0c7e523cec3b44c98e0b0cedea19ae30 P fb0783328ad3442fba39f260adf25308: Neither blocks nor log segments found. Creating new log.
I20260620 06:07:14.544377 19047 log.cc:824] T 0c7e523cec3b44c98e0b0cedea19ae30 P fb0783328ad3442fba39f260adf25308: Log is configured to *not* fsync() on all Append() calls
I20260620 06:07:14.544742 19045 tablet_bootstrap.cc:492] T 0c7e523cec3b44c98e0b0cedea19ae30 P f569ba0a0ea74d078dabe54eb42b9e21: No bootstrap required, opened a new log
I20260620 06:07:14.544922 19045 ts_tablet_manager.cc:1397] T 0c7e523cec3b44c98e0b0cedea19ae30 P f569ba0a0ea74d078dabe54eb42b9e21: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260620 06:07:14.545392 19046 tablet_bootstrap.cc:492] T 0c7e523cec3b44c98e0b0cedea19ae30 P d4b3fbe7b8664c968f75cc26b72410d7: No bootstrap required, opened a new log
I20260620 06:07:14.545562 19046 ts_tablet_manager.cc:1397] T 0c7e523cec3b44c98e0b0cedea19ae30 P d4b3fbe7b8664c968f75cc26b72410d7: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.003s
I20260620 06:07:14.545967 19047 tablet_bootstrap.cc:492] T 0c7e523cec3b44c98e0b0cedea19ae30 P fb0783328ad3442fba39f260adf25308: No bootstrap required, opened a new log
I20260620 06:07:14.546147 19047 ts_tablet_manager.cc:1397] T 0c7e523cec3b44c98e0b0cedea19ae30 P fb0783328ad3442fba39f260adf25308: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.002s
I20260620 06:07:14.549801 19047 raft_consensus.cc:348] T 0c7e523cec3b44c98e0b0cedea19ae30 P fb0783328ad3442fba39f260adf25308 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f569ba0a0ea74d078dabe54eb42b9e21" member_type: VOTER last_known_addr { host: "127.11.102.2" port: 33643 } } peers { permanent_uuid: "d4b3fbe7b8664c968f75cc26b72410d7" member_type: VOTER last_known_addr { host: "127.11.102.3" port: 36875 } } peers { permanent_uuid: "fb0783328ad3442fba39f260adf25308" member_type: VOTER last_known_addr { host: "127.11.102.1" port: 37681 } }
I20260620 06:07:14.549810 19045 raft_consensus.cc:348] T 0c7e523cec3b44c98e0b0cedea19ae30 P f569ba0a0ea74d078dabe54eb42b9e21 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f569ba0a0ea74d078dabe54eb42b9e21" member_type: VOTER last_known_addr { host: "127.11.102.2" port: 33643 } } peers { permanent_uuid: "d4b3fbe7b8664c968f75cc26b72410d7" member_type: VOTER last_known_addr { host: "127.11.102.3" port: 36875 } } peers { permanent_uuid: "fb0783328ad3442fba39f260adf25308" member_type: VOTER last_known_addr { host: "127.11.102.1" port: 37681 } }
I20260620 06:07:14.550047 19047 raft_consensus.cc:374] T 0c7e523cec3b44c98e0b0cedea19ae30 P fb0783328ad3442fba39f260adf25308 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 06:07:14.550063 19045 raft_consensus.cc:374] T 0c7e523cec3b44c98e0b0cedea19ae30 P f569ba0a0ea74d078dabe54eb42b9e21 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 06:07:14.550108 19047 raft_consensus.cc:729] T 0c7e523cec3b44c98e0b0cedea19ae30 P fb0783328ad3442fba39f260adf25308 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fb0783328ad3442fba39f260adf25308, State: Initialized, Role: FOLLOWER
I20260620 06:07:14.550113 19045 raft_consensus.cc:729] T 0c7e523cec3b44c98e0b0cedea19ae30 P f569ba0a0ea74d078dabe54eb42b9e21 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f569ba0a0ea74d078dabe54eb42b9e21, State: Initialized, Role: FOLLOWER
I20260620 06:07:14.550276 19046 raft_consensus.cc:348] T 0c7e523cec3b44c98e0b0cedea19ae30 P d4b3fbe7b8664c968f75cc26b72410d7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f569ba0a0ea74d078dabe54eb42b9e21" member_type: VOTER last_known_addr { host: "127.11.102.2" port: 33643 } } peers { permanent_uuid: "d4b3fbe7b8664c968f75cc26b72410d7" member_type: VOTER last_known_addr { host: "127.11.102.3" port: 36875 } } peers { permanent_uuid: "fb0783328ad3442fba39f260adf25308" member_type: VOTER last_known_addr { host: "127.11.102.1" port: 37681 } }
I20260620 06:07:14.550491 19046 raft_consensus.cc:374] T 0c7e523cec3b44c98e0b0cedea19ae30 P d4b3fbe7b8664c968f75cc26b72410d7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 06:07:14.550555 19046 raft_consensus.cc:729] T 0c7e523cec3b44c98e0b0cedea19ae30 P d4b3fbe7b8664c968f75cc26b72410d7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d4b3fbe7b8664c968f75cc26b72410d7, State: Initialized, Role: FOLLOWER
I20260620 06:07:14.550572 19045 consensus_queue.cc:260] T 0c7e523cec3b44c98e0b0cedea19ae30 P f569ba0a0ea74d078dabe54eb42b9e21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f569ba0a0ea74d078dabe54eb42b9e21" member_type: VOTER last_known_addr { host: "127.11.102.2" port: 33643 } } peers { permanent_uuid: "d4b3fbe7b8664c968f75cc26b72410d7" member_type: VOTER last_known_addr { host: "127.11.102.3" port: 36875 } } peers { permanent_uuid: "fb0783328ad3442fba39f260adf25308" member_type: VOTER last_known_addr { host: "127.11.102.1" port: 37681 } }
I20260620 06:07:14.550565 19047 consensus_queue.cc:260] T 0c7e523cec3b44c98e0b0cedea19ae30 P fb0783328ad3442fba39f260adf25308 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f569ba0a0ea74d078dabe54eb42b9e21" member_type: VOTER last_known_addr { host: "127.11.102.2" port: 33643 } } peers { permanent_uuid: "d4b3fbe7b8664c968f75cc26b72410d7" member_type: VOTER last_known_addr { host: "127.11.102.3" port: 36875 } } peers { permanent_uuid: "fb0783328ad3442fba39f260adf25308" member_type: VOTER last_known_addr { host: "127.11.102.1" port: 37681 } }
I20260620 06:07:14.550962 19046 consensus_queue.cc:260] T 0c7e523cec3b44c98e0b0cedea19ae30 P d4b3fbe7b8664c968f75cc26b72410d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f569ba0a0ea74d078dabe54eb42b9e21" member_type: VOTER last_known_addr { host: "127.11.102.2" port: 33643 } } peers { permanent_uuid: "d4b3fbe7b8664c968f75cc26b72410d7" member_type: VOTER last_known_addr { host: "127.11.102.3" port: 36875 } } peers { permanent_uuid: "fb0783328ad3442fba39f260adf25308" member_type: VOTER last_known_addr { host: "127.11.102.1" port: 37681 } }
I20260620 06:07:14.551517 19045 ts_tablet_manager.cc:1428] T 0c7e523cec3b44c98e0b0cedea19ae30 P f569ba0a0ea74d078dabe54eb42b9e21: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260620 06:07:14.551519 19047 ts_tablet_manager.cc:1428] T 0c7e523cec3b44c98e0b0cedea19ae30 P fb0783328ad3442fba39f260adf25308: Time spent starting tablet: real 0.005s user 0.004s sys 0.002s
I20260620 06:07:14.551801 19046 ts_tablet_manager.cc:1428] T 0c7e523cec3b44c98e0b0cedea19ae30 P d4b3fbe7b8664c968f75cc26b72410d7: Time spent starting tablet: real 0.006s user 0.005s sys 0.002s
I20260620 06:07:14.551858 18876 heartbeater.cc:503] Master 127.11.102.62:44777 was elected leader, sending a full tablet report...
I20260620 06:07:14.552520 18739 heartbeater.cc:503] Master 127.11.102.62:44777 was elected leader, sending a full tablet report...
I20260620 06:07:14.556406 19013 heartbeater.cc:503] Master 127.11.102.62:44777 was elected leader, sending a full tablet report...
W20260620 06:07:14.592495 18741 tablet.cc:2369] T 0c7e523cec3b44c98e0b0cedea19ae30 P fb0783328ad3442fba39f260adf25308: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 06:07:14.731422 19015 tablet.cc:2369] T 0c7e523cec3b44c98e0b0cedea19ae30 P d4b3fbe7b8664c968f75cc26b72410d7: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 06:07:14.741201 19051 raft_consensus.cc:482] T 0c7e523cec3b44c98e0b0cedea19ae30 P f569ba0a0ea74d078dabe54eb42b9e21 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 06:07:14.741406 19051 raft_consensus.cc:504] T 0c7e523cec3b44c98e0b0cedea19ae30 P f569ba0a0ea74d078dabe54eb42b9e21 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f569ba0a0ea74d078dabe54eb42b9e21" member_type: VOTER last_known_addr { host: "127.11.102.2" port: 33643 } } peers { permanent_uuid: "d4b3fbe7b8664c968f75cc26b72410d7" member_type: VOTER last_known_addr { host: "127.11.102.3" port: 36875 } } peers { permanent_uuid: "fb0783328ad3442fba39f260adf25308" member_type: VOTER last_known_addr { host: "127.11.102.1" port: 37681 } }
I20260620 06:07:14.742488 19051 leader_election.cc:288] T 0c7e523cec3b44c98e0b0cedea19ae30 P f569ba0a0ea74d078dabe54eb42b9e21 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d4b3fbe7b8664c968f75cc26b72410d7 (127.11.102.3:36875), fb0783328ad3442fba39f260adf25308 (127.11.102.1:37681)
I20260620 06:07:14.746335 19044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:07:14.742894 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.102.2:47985 (local address 127.11.102.1:37681)
0620 06:07:14.743034 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0620 06:07:14.743038 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 06:07:14.743203 (+ 165us) server_negotiation.cc:407] Connection header received
0620 06:07:14.743331 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:07:14.743333 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:07:14.743402 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:07:14.743479 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 06:07:14.743757 (+ 278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:07:14.744228 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:07:14.745356 (+ 1128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:07:14.745744 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:07:14.745869 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:07:14.745923 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0620 06:07:14.746154 (+ 231us) server_negotiation.cc:299] Negotiation successful
0620 06:07:14.746219 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260620 06:07:14.746337 19043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:07:14.742885 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.102.2:46547 (local address 127.11.102.3:36875)
0620 06:07:14.743027 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0620 06:07:14.743032 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 06:07:14.743051 (+ 19us) server_negotiation.cc:407] Connection header received
0620 06:07:14.743218 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:07:14.743222 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:07:14.743289 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:07:14.743361 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 06:07:14.743688 (+ 327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:07:14.744229 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:07:14.745349 (+ 1120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:07:14.745755 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:07:14.745880 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:07:14.745933 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0620 06:07:14.746120 (+ 187us) server_negotiation.cc:299] Negotiation successful
0620 06:07:14.746198 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260620 06:07:14.746357 18879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:07:14.742775 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.102.1:37681 (local address 127.11.102.2:47985)
0620 06:07:14.743082 (+ 307us) negotiation.cc:107] Waiting for socket to connect
0620 06:07:14.743101 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 06:07:14.743217 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:07:14.743502 (+ 285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:07:14.743505 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:07:14.743519 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 06:07:14.743652 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:07:14.743669 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:07:14.744352 (+ 683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:07:14.744355 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:07:14.745233 (+ 878us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:07:14.745242 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:07:14.745861 (+ 619us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:07:14.745865 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:07:14.745960 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:07:14.746039 (+ 79us) client_negotiation.cc:769] Sending connection context
0620 06:07:14.746127 (+ 88us) client_negotiation.cc:240] Negotiation successful
0620 06:07:14.746198 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":245,"mutex_wait_us":268}
I20260620 06:07:14.746361 18880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:07:14.742775 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.102.3:36875 (local address 127.11.102.2:46547)
0620 06:07:14.742937 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0620 06:07:14.742956 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 06:07:14.743069 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:07:14.743391 (+ 322us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:07:14.743395 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:07:14.743411 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 06:07:14.743550 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:07:14.743558 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:07:14.744337 (+ 779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:07:14.744343 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:07:14.745230 (+ 887us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:07:14.745238 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:07:14.745883 (+ 645us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:07:14.745886 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:07:14.746012 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:07:14.746028 (+ 16us) client_negotiation.cc:769] Sending connection context
0620 06:07:14.746086 (+ 58us) client_negotiation.cc:240] Negotiation successful
0620 06:07:14.746182 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":59,"mutex_wait_us":36}
I20260620 06:07:14.747056 18692 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c7e523cec3b44c98e0b0cedea19ae30" candidate_uuid: "f569ba0a0ea74d078dabe54eb42b9e21" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fb0783328ad3442fba39f260adf25308" is_pre_election: true
I20260620 06:07:14.747097 18966 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c7e523cec3b44c98e0b0cedea19ae30" candidate_uuid: "f569ba0a0ea74d078dabe54eb42b9e21" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4b3fbe7b8664c968f75cc26b72410d7" is_pre_election: true
I20260620 06:07:14.747365 18966 raft_consensus.cc:2454] T 0c7e523cec3b44c98e0b0cedea19ae30 P d4b3fbe7b8664c968f75cc26b72410d7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f569ba0a0ea74d078dabe54eb42b9e21 in term 0.
I20260620 06:07:14.747366 18692 raft_consensus.cc:2454] T 0c7e523cec3b44c98e0b0cedea19ae30 P fb0783328ad3442fba39f260adf25308 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f569ba0a0ea74d078dabe54eb42b9e21 in term 0.
I20260620 06:07:14.747844 18764 leader_election.cc:302] T 0c7e523cec3b44c98e0b0cedea19ae30 P f569ba0a0ea74d078dabe54eb42b9e21 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: d4b3fbe7b8664c968f75cc26b72410d7, f569ba0a0ea74d078dabe54eb42b9e21, fb0783328ad3442fba39f260adf25308; no voters:
I20260620 06:07:14.748090 19051 raft_consensus.cc:2781] T 0c7e523cec3b44c98e0b0cedea19ae30 P f569ba0a0ea74d078dabe54eb42b9e21 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 06:07:14.748178 19051 raft_consensus.cc:482] T 0c7e523cec3b44c98e0b0cedea19ae30 P f569ba0a0ea74d078dabe54eb42b9e21 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 06:07:14.748237 19051 raft_consensus.cc:3037] T 0c7e523cec3b44c98e0b0cedea19ae30 P f569ba0a0ea74d078dabe54eb42b9e21 [term 0 FOLLOWER]: Advancing to term 1
I20260620 06:07:14.749233 19051 raft_consensus.cc:504] T 0c7e523cec3b44c98e0b0cedea19ae30 P f569ba0a0ea74d078dabe54eb42b9e21 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f569ba0a0ea74d078dabe54eb42b9e21" member_type: VOTER last_known_addr { host: "127.11.102.2" port: 33643 } } peers { permanent_uuid: "d4b3fbe7b8664c968f75cc26b72410d7" member_type: VOTER last_known_addr { host: "127.11.102.3" port: 36875 } } peers { permanent_uuid: "fb0783328ad3442fba39f260adf25308" member_type: VOTER last_known_addr { host: "127.11.102.1" port: 37681 } }
I20260620 06:07:14.749576 19051 leader_election.cc:288] T 0c7e523cec3b44c98e0b0cedea19ae30 P f569ba0a0ea74d078dabe54eb42b9e21 [CANDIDATE]: Term 1 election: Requested vote from peers d4b3fbe7b8664c968f75cc26b72410d7 (127.11.102.3:36875), fb0783328ad3442fba39f260adf25308 (127.11.102.1:37681)
I20260620 06:07:14.749912 18692 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c7e523cec3b44c98e0b0cedea19ae30" candidate_uuid: "f569ba0a0ea74d078dabe54eb42b9e21" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fb0783328ad3442fba39f260adf25308"
I20260620 06:07:14.749913 18966 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c7e523cec3b44c98e0b0cedea19ae30" candidate_uuid: "f569ba0a0ea74d078dabe54eb42b9e21" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4b3fbe7b8664c968f75cc26b72410d7"
I20260620 06:07:14.750030 18966 raft_consensus.cc:3037] T 0c7e523cec3b44c98e0b0cedea19ae30 P d4b3fbe7b8664c968f75cc26b72410d7 [term 0 FOLLOWER]: Advancing to term 1
I20260620 06:07:14.750030 18692 raft_consensus.cc:3037] T 0c7e523cec3b44c98e0b0cedea19ae30 P fb0783328ad3442fba39f260adf25308 [term 0 FOLLOWER]: Advancing to term 1
I20260620 06:07:14.751230 18692 raft_consensus.cc:2454] T 0c7e523cec3b44c98e0b0cedea19ae30 P fb0783328ad3442fba39f260adf25308 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f569ba0a0ea74d078dabe54eb42b9e21 in term 1.
I20260620 06:07:14.751230 18966 raft_consensus.cc:2454] T 0c7e523cec3b44c98e0b0cedea19ae30 P d4b3fbe7b8664c968f75cc26b72410d7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f569ba0a0ea74d078dabe54eb42b9e21 in term 1.
I20260620 06:07:14.751507 18763 leader_election.cc:302] T 0c7e523cec3b44c98e0b0cedea19ae30 P f569ba0a0ea74d078dabe54eb42b9e21 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: d4b3fbe7b8664c968f75cc26b72410d7, f569ba0a0ea74d078dabe54eb42b9e21, fb0783328ad3442fba39f260adf25308; no voters:
I20260620 06:07:14.751719 19051 raft_consensus.cc:2781] T 0c7e523cec3b44c98e0b0cedea19ae30 P f569ba0a0ea74d078dabe54eb42b9e21 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 06:07:14.751940 19051 raft_consensus.cc:686] T 0c7e523cec3b44c98e0b0cedea19ae30 P f569ba0a0ea74d078dabe54eb42b9e21 [term 1 LEADER]: Becoming Leader. State: Replica: f569ba0a0ea74d078dabe54eb42b9e21, State: Running, Role: LEADER
I20260620 06:07:14.752287 19051 consensus_queue.cc:237] T 0c7e523cec3b44c98e0b0cedea19ae30 P f569ba0a0ea74d078dabe54eb42b9e21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f569ba0a0ea74d078dabe54eb42b9e21" member_type: VOTER last_known_addr { host: "127.11.102.2" port: 33643 } } peers { permanent_uuid: "d4b3fbe7b8664c968f75cc26b72410d7" member_type: VOTER last_known_addr { host: "127.11.102.3" port: 36875 } } peers { permanent_uuid: "fb0783328ad3442fba39f260adf25308" member_type: VOTER last_known_addr { host: "127.11.102.1" port: 37681 } }
I20260620 06:07:14.755838 17774 catalog_manager.cc:5495] T 0c7e523cec3b44c98e0b0cedea19ae30 P f569ba0a0ea74d078dabe54eb42b9e21 reported cstate change: term changed from 0 to 1, leader changed from <none> to f569ba0a0ea74d078dabe54eb42b9e21 (127.11.102.2). New cstate: current_term: 1 leader_uuid: "f569ba0a0ea74d078dabe54eb42b9e21" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f569ba0a0ea74d078dabe54eb42b9e21" member_type: VOTER last_known_addr { host: "127.11.102.2" port: 33643 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d4b3fbe7b8664c968f75cc26b72410d7" member_type: VOTER last_known_addr { host: "127.11.102.3" port: 36875 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fb0783328ad3442fba39f260adf25308" member_type: VOTER last_known_addr { host: "127.11.102.1" port: 37681 } health_report { overall_health: UNKNOWN } } }
W20260620 06:07:14.792639 18878 tablet.cc:2369] T 0c7e523cec3b44c98e0b0cedea19ae30 P f569ba0a0ea74d078dabe54eb42b9e21: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 06:07:14.817693 19040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:07:14.814182 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59310 (local address 127.11.102.2:33643)
0620 06:07:14.814328 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0620 06:07:14.814333 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 06:07:14.814347 (+ 14us) server_negotiation.cc:407] Connection header received
0620 06:07:14.814456 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:07:14.814459 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:07:14.814510 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:07:14.814630 (+ 120us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 06:07:14.814862 (+ 232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:07:14.815365 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:07:14.816242 (+ 877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:07:14.816459 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:07:14.816573 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:07:14.816871 (+ 298us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 06:07:14.817297 (+ 426us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 06:07:14.817441 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0620 06:07:14.817485 (+ 44us) server_negotiation.cc:299] Negotiation successful
0620 06:07:14.817541 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260620 06:07:14.823532 18966 raft_consensus.cc:1261] T 0c7e523cec3b44c98e0b0cedea19ae30 P d4b3fbe7b8664c968f75cc26b72410d7 [term 1 FOLLOWER]: Refusing update from remote peer f569ba0a0ea74d078dabe54eb42b9e21: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 06:07:14.823532 18692 raft_consensus.cc:1261] T 0c7e523cec3b44c98e0b0cedea19ae30 P fb0783328ad3442fba39f260adf25308 [term 1 FOLLOWER]: Refusing update from remote peer f569ba0a0ea74d078dabe54eb42b9e21: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 06:07:14.824322 19051 consensus_queue.cc:1035] T 0c7e523cec3b44c98e0b0cedea19ae30 P f569ba0a0ea74d078dabe54eb42b9e21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fb0783328ad3442fba39f260adf25308" member_type: VOTER last_known_addr { host: "127.11.102.1" port: 37681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 06:07:14.824493 19054 consensus_queue.cc:1035] T 0c7e523cec3b44c98e0b0cedea19ae30 P f569ba0a0ea74d078dabe54eb42b9e21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4b3fbe7b8664c968f75cc26b72410d7" member_type: VOTER last_known_addr { host: "127.11.102.3" port: 36875 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 06:07:14.832304 19059 mvcc.cc:205] Tried to move back new op lower bound from 7298808360230277120 to 7298808359949352960. Current Snapshot: MvccSnapshot[applied={T|T < 7298808360230277120}]
I20260620 06:07:14.841089 17773 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54674:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 06:07:14.841308 17773 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54674:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 06:07:14.845628 17773 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d7d4500dc38a48a1858293d72d1fcfc2: Sending DeleteTablet for 3 replicas of tablet 0c7e523cec3b44c98e0b0cedea19ae30
I20260620 06:07:14.846398 18946 tablet_service.cc:1514] Processing DeleteTablet for tablet 0c7e523cec3b44c98e0b0cedea19ae30 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 06:07:14 UTC) from {username='oryx', principal='oryx/127.11.102.62@KRBTEST.COM'} at 127.0.0.1:58686
I20260620 06:07:14.846434 18672 tablet_service.cc:1514] Processing DeleteTablet for tablet 0c7e523cec3b44c98e0b0cedea19ae30 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 06:07:14 UTC) from {username='oryx', principal='oryx/127.11.102.62@KRBTEST.COM'} at 127.0.0.1:52236
I20260620 06:07:14.846465 18809 tablet_service.cc:1514] Processing DeleteTablet for tablet 0c7e523cec3b44c98e0b0cedea19ae30 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 06:07:14 UTC) from {username='oryx', principal='oryx/127.11.102.62@KRBTEST.COM'} at 127.0.0.1:59304
I20260620 06:07:14.847528 19066 tablet_replica.cc:330] T 0c7e523cec3b44c98e0b0cedea19ae30 P fb0783328ad3442fba39f260adf25308: stopping tablet replica
I20260620 06:07:14.847857 19066 raft_consensus.cc:2229] T 0c7e523cec3b44c98e0b0cedea19ae30 P fb0783328ad3442fba39f260adf25308 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 06:07:14.848163 19066 raft_consensus.cc:2258] T 0c7e523cec3b44c98e0b0cedea19ae30 P fb0783328ad3442fba39f260adf25308 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 06:07:14.848780 11672 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9HMp43/build/debug/bin/kudu with pid 17742
I20260620 06:07:14.850376 19066 ts_tablet_manager.cc:1905] T 0c7e523cec3b44c98e0b0cedea19ae30 P fb0783328ad3442fba39f260adf25308: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 06:07:14.853046 19066 ts_tablet_manager.cc:1918] T 0c7e523cec3b44c98e0b0cedea19ae30 P fb0783328ad3442fba39f260adf25308: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 06:07:14.853157 19066 log.cc:1196] T 0c7e523cec3b44c98e0b0cedea19ae30 P fb0783328ad3442fba39f260adf25308: Deleting WAL directory at /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935603120653-11672-0/minicluster-data/ts-0/wal/wals/0c7e523cec3b44c98e0b0cedea19ae30
I20260620 06:07:14.853469 19066 ts_tablet_manager.cc:1939] T 0c7e523cec3b44c98e0b0cedea19ae30 P fb0783328ad3442fba39f260adf25308: Deleting consensus metadata
I20260620 06:07:14.854725 19067 tablet_replica.cc:330] T 0c7e523cec3b44c98e0b0cedea19ae30 P f569ba0a0ea74d078dabe54eb42b9e21: stopping tablet replica
I20260620 06:07:14.855057 19067 raft_consensus.cc:2229] T 0c7e523cec3b44c98e0b0cedea19ae30 P f569ba0a0ea74d078dabe54eb42b9e21 [term 1 LEADER]: Raft consensus shutting down.
I20260620 06:07:14.855444 19067 raft_consensus.cc:2258] T 0c7e523cec3b44c98e0b0cedea19ae30 P f569ba0a0ea74d078dabe54eb42b9e21 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260620 06:07:14.856112 18625 connection.cc:535] server connection from 127.0.0.1:52236 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260620 06:07:14.857601 19067 ts_tablet_manager.cc:1905] T 0c7e523cec3b44c98e0b0cedea19ae30 P f569ba0a0ea74d078dabe54eb42b9e21: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 06:07:14.860301 19067 ts_tablet_manager.cc:1918] T 0c7e523cec3b44c98e0b0cedea19ae30 P f569ba0a0ea74d078dabe54eb42b9e21: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 06:07:14.860411 19067 log.cc:1196] T 0c7e523cec3b44c98e0b0cedea19ae30 P f569ba0a0ea74d078dabe54eb42b9e21: Deleting WAL directory at /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935603120653-11672-0/minicluster-data/ts-1/wal/wals/0c7e523cec3b44c98e0b0cedea19ae30
I20260620 06:07:14.860778 19067 ts_tablet_manager.cc:1939] T 0c7e523cec3b44c98e0b0cedea19ae30 P f569ba0a0ea74d078dabe54eb42b9e21: Deleting consensus metadata
W20260620 06:07:14.861752 18763 connection.cc:441] server connection from 127.0.0.1:59304 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59304 (request call id 1) could send its response
I20260620 06:07:14.863113 19068 tablet_replica.cc:330] T 0c7e523cec3b44c98e0b0cedea19ae30 P d4b3fbe7b8664c968f75cc26b72410d7: stopping tablet replica
I20260620 06:07:14.863430 19068 raft_consensus.cc:2229] T 0c7e523cec3b44c98e0b0cedea19ae30 P d4b3fbe7b8664c968f75cc26b72410d7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 06:07:14.865962 19068 raft_consensus.cc:2258] T 0c7e523cec3b44c98e0b0cedea19ae30 P d4b3fbe7b8664c968f75cc26b72410d7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 06:07:14.867254 19023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:07:14.847848 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52514 (local address 127.11.102.61:39977)
0620 06:07:14.847951 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0620 06:07:14.847955 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 06:07:14.854851 (+ 6896us) server_negotiation.cc:407] Connection header received
0620 06:07:14.855049 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:07:14.855055 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:07:14.855121 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:07:14.855204 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:07:14.857045 (+ 1841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:07:14.857881 (+ 836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:07:14.859067 (+ 1186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:07:14.859408 (+ 341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:07:14.859498 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:07:14.861720 (+ 2222us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:07:14.861726 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:07:14.861733 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:07:14.861764 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:07:14.863455 (+ 1691us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:07:14.864270 (+ 815us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:07:14.864275 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:07:14.864276 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:07:14.864363 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:07:14.864970 (+ 607us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:07:14.864975 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:07:14.864977 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:07:14.865194 (+ 217us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:07:14.865573 (+ 379us) server_negotiation.cc:1035] Waiting for connection context
0620 06:07:14.866951 (+ 1378us) server_negotiation.cc:299] Negotiation successful
0620 06:07:14.867074 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260620 06:07:14.867965 19068 ts_tablet_manager.cc:1905] T 0c7e523cec3b44c98e0b0cedea19ae30 P d4b3fbe7b8664c968f75cc26b72410d7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 06:07:14.868804 18498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:07:14.858528 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52524 (local address 127.11.102.61:39977)
0620 06:07:14.858636 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0620 06:07:14.858640 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 06:07:14.859477 (+ 837us) server_negotiation.cc:407] Connection header received
0620 06:07:14.859787 (+ 310us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:07:14.859791 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:07:14.859835 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:07:14.859889 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:07:14.860893 (+ 1004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:07:14.861663 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:07:14.862895 (+ 1232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:07:14.863287 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:07:14.863373 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:07:14.865001 (+ 1628us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:07:14.865006 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:07:14.865012 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:07:14.865037 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:07:14.866741 (+ 1704us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:07:14.867269 (+ 528us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:07:14.867273 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:07:14.867275 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:07:14.867353 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:07:14.867799 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:07:14.867802 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:07:14.867804 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:07:14.868027 (+ 223us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:07:14.868121 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0620 06:07:14.868507 (+ 386us) server_negotiation.cc:299] Negotiation successful
0620 06:07:14.868617 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260620 06:07:14.870069 18883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:07:14.858893 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56128 (local address 127.11.102.60:41691)
0620 06:07:14.859016 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0620 06:07:14.859022 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 06:07:14.859678 (+ 656us) server_negotiation.cc:407] Connection header received
0620 06:07:14.859848 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:07:14.859851 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:07:14.859898 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:07:14.859952 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:07:14.861678 (+ 1726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:07:14.862469 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:07:14.863727 (+ 1258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:07:14.864077 (+ 350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:07:14.864158 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:07:14.865827 (+ 1669us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:07:14.865830 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:07:14.865833 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:07:14.865858 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:07:14.867567 (+ 1709us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:07:14.868406 (+ 839us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:07:14.868410 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:07:14.868412 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:07:14.868493 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:07:14.868776 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:07:14.868779 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:07:14.868781 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:07:14.868963 (+ 182us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:07:14.869052 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0620 06:07:14.869781 (+ 729us) server_negotiation.cc:299] Negotiation successful
0620 06:07:14.869925 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260620 06:07:14.870846 19068 ts_tablet_manager.cc:1918] T 0c7e523cec3b44c98e0b0cedea19ae30 P d4b3fbe7b8664c968f75cc26b72410d7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 06:07:14.870942 19068 log.cc:1196] T 0c7e523cec3b44c98e0b0cedea19ae30 P d4b3fbe7b8664c968f75cc26b72410d7: Deleting WAL directory at /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935603120653-11672-0/minicluster-data/ts-2/wal/wals/0c7e523cec3b44c98e0b0cedea19ae30
I20260620 06:07:14.871294 19068 ts_tablet_manager.cc:1939] T 0c7e523cec3b44c98e0b0cedea19ae30 P d4b3fbe7b8664c968f75cc26b72410d7: Deleting consensus metadata
W20260620 06:07:14.872593 18902 connection.cc:441] server connection from 127.0.0.1:58686 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58686 (request call id 1) could send its response
I20260620 06:07:14.875373 18745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:07:14.848749 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56126 (local address 127.11.102.60:41691)
0620 06:07:14.848870 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0620 06:07:14.848876 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 06:07:14.866742 (+ 17866us) server_negotiation.cc:407] Connection header received
0620 06:07:14.868314 (+ 1572us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:07:14.868317 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:07:14.868372 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:07:14.868427 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:07:14.869179 (+ 752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:07:14.869863 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:07:14.870785 (+ 922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:07:14.871084 (+ 299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:07:14.871205 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:07:14.872686 (+ 1481us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:07:14.872691 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:07:14.872694 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:07:14.872720 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:07:14.873926 (+ 1206us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:07:14.874362 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:07:14.874364 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:07:14.874365 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:07:14.874427 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:07:14.874713 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:07:14.874715 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:07:14.874716 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:07:14.874861 (+ 145us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:07:14.874952 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0620 06:07:14.875156 (+ 204us) server_negotiation.cc:299] Negotiation successful
0620 06:07:14.875232 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260620 06:07:15.836403 19077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:07:15.835932 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.102.62:44777 (local address 127.11.102.3:39191)
0620 06:07:15.836178 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0620 06:07:15.836314 (+ 136us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.102.62:44777: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":168,"thread_start_us":66,"threads_started":1}
W20260620 06:07:15.836725 19013 heartbeater.cc:644] Failed to heartbeat to 127.11.102.62:44777 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.11.102.62:44777: connect: Connection refused (error 111)
I20260620 06:07:15.840168 19078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:07:15.839765 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.102.62:44777 (local address 127.11.102.1:58195)
0620 06:07:15.839984 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0620 06:07:15.840071 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.102.62:44777: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":59,"threads_started":1}
W20260620 06:07:15.840562 18739 heartbeater.cc:644] Failed to heartbeat to 127.11.102.62:44777 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.11.102.62:44777: connect: Connection refused (error 111)
I20260620 06:07:15.841952 19079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:07:15.841612 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.102.62:44777 (local address 127.11.102.2:54439)
0620 06:07:15.841817 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0620 06:07:15.841881 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.102.62:44777: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":59,"threads_started":1}
W20260620 06:07:15.842199 18876 heartbeater.cc:644] Failed to heartbeat to 127.11.102.62:44777 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.11.102.62:44777: connect: Connection refused (error 111)
I20260620 06:07:16.346470 19080 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0a9e7156fd5c4d399486dc5932b407b2 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d7d4500dc38a48a1858293d72d1fcfc2)
I20260620 06:07:16.346750 19080 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0a9e7156fd5c4d399486dc5932b407b2 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "d7d4500dc38a48a1858293d72d1fcfc2" member_type: VOTER last_known_addr { host: "127.11.102.62" port: 44777 } } peers { permanent_uuid: "0a9e7156fd5c4d399486dc5932b407b2" member_type: VOTER last_known_addr { host: "127.11.102.61" port: 39977 } attrs { promote: false } } peers { permanent_uuid: "ec929d712e20476684cecae76f307d8e" member_type: VOTER last_known_addr { host: "127.11.102.60" port: 41691 } attrs { promote: false } }
I20260620 06:07:16.346796 19081 raft_consensus.cc:482] T 00000000000000000000000000000000 P ec929d712e20476684cecae76f307d8e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d7d4500dc38a48a1858293d72d1fcfc2)
I20260620 06:07:16.346997 19081 raft_consensus.cc:504] T 00000000000000000000000000000000 P ec929d712e20476684cecae76f307d8e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "d7d4500dc38a48a1858293d72d1fcfc2" member_type: VOTER last_known_addr { host: "127.11.102.62" port: 44777 } } peers { permanent_uuid: "0a9e7156fd5c4d399486dc5932b407b2" member_type: VOTER last_known_addr { host: "127.11.102.61" port: 39977 } attrs { promote: false } } peers { permanent_uuid: "ec929d712e20476684cecae76f307d8e" member_type: VOTER last_known_addr { host: "127.11.102.60" port: 41691 } attrs { promote: false } }
I20260620 06:07:16.347870 19080 leader_election.cc:288] T 00000000000000000000000000000000 P 0a9e7156fd5c4d399486dc5932b407b2 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d7d4500dc38a48a1858293d72d1fcfc2 (127.11.102.62:44777), ec929d712e20476684cecae76f307d8e (127.11.102.60:41691)
I20260620 06:07:16.347918 19081 leader_election.cc:288] T 00000000000000000000000000000000 P ec929d712e20476684cecae76f307d8e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d7d4500dc38a48a1858293d72d1fcfc2 (127.11.102.62:44777), 0a9e7156fd5c4d399486dc5932b407b2 (127.11.102.61:39977)
I20260620 06:07:16.348423 19082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:07:16.348022 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.102.62:44777 (local address 127.0.0.1:55186)
0620 06:07:16.348231 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0620 06:07:16.348310 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.102.62:44777: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":56,"threads_started":1}
I20260620 06:07:16.348789 19083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:07:16.348017 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.102.62:44777 (local address 127.0.0.1:55180)
0620 06:07:16.348604 (+ 587us) negotiation.cc:107] Waiting for socket to connect
0620 06:07:16.348680 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.102.62:44777: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":516,"thread_start_us":81,"threads_started":1}
W20260620 06:07:16.348794 18544 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.11.102.62:44777: connect: Connection refused (error 111)
W20260620 06:07:16.348949 18361 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.11.102.62:44777: connect: Connection refused (error 111)
I20260620 06:07:16.349833 19083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:07:16.349607 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.102.62:44777 (local address 127.0.0.1:55202)
0620 06:07:16.349723 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0620 06:07:16.349776 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.102.62:44777: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260620 06:07:16.350036 18361 leader_election.cc:334] T 00000000000000000000000000000000 P 0a9e7156fd5c4d399486dc5932b407b2 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer d7d4500dc38a48a1858293d72d1fcfc2 (127.11.102.62:44777): Network error: Client connection negotiation failed: client connection to 127.11.102.62:44777: connect: Connection refused (error 111)
I20260620 06:07:16.350080 19082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:07:16.349857 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.102.62:44777 (local address 127.0.0.1:55206)
0620 06:07:16.349980 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0620 06:07:16.350023 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.102.62:44777: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
W20260620 06:07:16.350189 18544 leader_election.cc:334] T 00000000000000000000000000000000 P ec929d712e20476684cecae76f307d8e [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer d7d4500dc38a48a1858293d72d1fcfc2 (127.11.102.62:44777): Network error: Client connection negotiation failed: client connection to 127.11.102.62:44777: connect: Connection refused (error 111)
I20260620 06:07:16.352342 19085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:07:16.348310 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52538 (local address 127.11.102.61:39977)
0620 06:07:16.348729 (+ 419us) server_negotiation.cc:206] Beginning negotiation
0620 06:07:16.348734 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 06:07:16.348749 (+ 15us) server_negotiation.cc:407] Connection header received
0620 06:07:16.348795 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:07:16.348798 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:07:16.348955 (+ 157us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:07:16.349222 (+ 267us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 06:07:16.349498 (+ 276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:07:16.350273 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:07:16.351324 (+ 1051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:07:16.351780 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:07:16.351894 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:07:16.351950 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0620 06:07:16.352161 (+ 211us) server_negotiation.cc:299] Negotiation successful
0620 06:07:16.352227 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":354,"thread_start_us":64,"threads_started":1}
I20260620 06:07:16.352349 19084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:07:16.348173 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.102.61:39977 (local address 127.0.0.1:52538)
0620 06:07:16.348428 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0620 06:07:16.348443 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 06:07:16.348546 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:07:16.349231 (+ 685us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:07:16.349234 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:07:16.349253 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 06:07:16.349357 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:07:16.349366 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:07:16.350402 (+ 1036us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:07:16.350406 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:07:16.351188 (+ 782us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:07:16.351196 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:07:16.351900 (+ 704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:07:16.351903 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:07:16.352034 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:07:16.352060 (+ 26us) client_negotiation.cc:769] Sending connection context
0620 06:07:16.352147 (+ 87us) client_negotiation.cc:240] Negotiation successful
0620 06:07:16.352218 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":182,"mutex_wait_us":23,"thread_start_us":51,"threads_started":1}
I20260620 06:07:16.352348 19087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:07:16.348608 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56132 (local address 127.11.102.60:41691)
0620 06:07:16.349068 (+ 460us) server_negotiation.cc:206] Beginning negotiation
0620 06:07:16.349073 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 06:07:16.349097 (+ 24us) server_negotiation.cc:407] Connection header received
0620 06:07:16.349140 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:07:16.349145 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:07:16.349198 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:07:16.349543 (+ 345us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 06:07:16.349640 (+ 97us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:07:16.350373 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:07:16.351360 (+ 987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:07:16.351835 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:07:16.351955 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:07:16.352086 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0620 06:07:16.352163 (+ 77us) server_negotiation.cc:299] Negotiation successful
0620 06:07:16.352225 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":45,"server-negotiator.queue_time_us":397,"thread_start_us":68,"threads_started":1}
I20260620 06:07:16.352348 19086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:07:16.348348 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.102.60:41691 (local address 127.0.0.1:56132)
0620 06:07:16.348790 (+ 442us) negotiation.cc:107] Waiting for socket to connect
0620 06:07:16.348804 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 06:07:16.348890 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:07:16.349319 (+ 429us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:07:16.349322 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:07:16.349339 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 06:07:16.349435 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:07:16.349442 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:07:16.350485 (+ 1043us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:07:16.350492 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:07:16.351241 (+ 749us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:07:16.351248 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:07:16.351946 (+ 698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:07:16.351950 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:07:16.352048 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:07:16.352070 (+ 22us) client_negotiation.cc:769] Sending connection context
0620 06:07:16.352151 (+ 81us) client_negotiation.cc:240] Negotiation successful
0620 06:07:16.352221 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":389,"thread_start_us":53,"threads_started":1}
I20260620 06:07:16.352890 18569 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "0a9e7156fd5c4d399486dc5932b407b2" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "ec929d712e20476684cecae76f307d8e" is_pre_election: true
I20260620 06:07:16.353089 18569 raft_consensus.cc:2454] T 00000000000000000000000000000000 P ec929d712e20476684cecae76f307d8e [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a9e7156fd5c4d399486dc5932b407b2 in term 1.
I20260620 06:07:16.353127 18381 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ec929d712e20476684cecae76f307d8e" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "0a9e7156fd5c4d399486dc5932b407b2" is_pre_election: true
I20260620 06:07:16.353334 18381 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 0a9e7156fd5c4d399486dc5932b407b2 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ec929d712e20476684cecae76f307d8e in term 1.
I20260620 06:07:16.353406 18361 leader_election.cc:302] T 00000000000000000000000000000000 P 0a9e7156fd5c4d399486dc5932b407b2 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a9e7156fd5c4d399486dc5932b407b2, ec929d712e20476684cecae76f307d8e; no voters: d7d4500dc38a48a1858293d72d1fcfc2
I20260620 06:07:16.353641 19080 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0a9e7156fd5c4d399486dc5932b407b2 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260620 06:07:16.353663 18545 leader_election.cc:302] T 00000000000000000000000000000000 P ec929d712e20476684cecae76f307d8e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a9e7156fd5c4d399486dc5932b407b2, ec929d712e20476684cecae76f307d8e; no voters: d7d4500dc38a48a1858293d72d1fcfc2
I20260620 06:07:16.353719 19080 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0a9e7156fd5c4d399486dc5932b407b2 [term 1 FOLLOWER]: Starting leader election (detected failure of leader d7d4500dc38a48a1858293d72d1fcfc2)
I20260620 06:07:16.353766 19080 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0a9e7156fd5c4d399486dc5932b407b2 [term 1 FOLLOWER]: Advancing to term 2
I20260620 06:07:16.354038 19081 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ec929d712e20476684cecae76f307d8e [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260620 06:07:16.354097 19081 raft_consensus.cc:482] T 00000000000000000000000000000000 P ec929d712e20476684cecae76f307d8e [term 1 FOLLOWER]: Starting leader election (detected failure of leader d7d4500dc38a48a1858293d72d1fcfc2)
I20260620 06:07:16.354136 19081 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ec929d712e20476684cecae76f307d8e [term 1 FOLLOWER]: Advancing to term 2
I20260620 06:07:16.355041 19080 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0a9e7156fd5c4d399486dc5932b407b2 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "d7d4500dc38a48a1858293d72d1fcfc2" member_type: VOTER last_known_addr { host: "127.11.102.62" port: 44777 } } peers { permanent_uuid: "0a9e7156fd5c4d399486dc5932b407b2" member_type: VOTER last_known_addr { host: "127.11.102.61" port: 39977 } attrs { promote: false } } peers { permanent_uuid: "ec929d712e20476684cecae76f307d8e" member_type: VOTER last_known_addr { host: "127.11.102.60" port: 41691 } attrs { promote: false } }
I20260620 06:07:16.355304 19081 raft_consensus.cc:504] T 00000000000000000000000000000000 P ec929d712e20476684cecae76f307d8e [term 2 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "d7d4500dc38a48a1858293d72d1fcfc2" member_type: VOTER last_known_addr { host: "127.11.102.62" port: 44777 } } peers { permanent_uuid: "0a9e7156fd5c4d399486dc5932b407b2" member_type: VOTER last_known_addr { host: "127.11.102.61" port: 39977 } attrs { promote: false } } peers { permanent_uuid: "ec929d712e20476684cecae76f307d8e" member_type: VOTER last_known_addr { host: "127.11.102.60" port: 41691 } attrs { promote: false } }
I20260620 06:07:16.355413 19080 leader_election.cc:288] T 00000000000000000000000000000000 P 0a9e7156fd5c4d399486dc5932b407b2 [CANDIDATE]: Term 2 election: Requested vote from peers d7d4500dc38a48a1858293d72d1fcfc2 (127.11.102.62:44777), ec929d712e20476684cecae76f307d8e (127.11.102.60:41691)
I20260620 06:07:16.355684 19081 leader_election.cc:288] T 00000000000000000000000000000000 P ec929d712e20476684cecae76f307d8e [CANDIDATE]: Term 2 election: Requested vote from peers d7d4500dc38a48a1858293d72d1fcfc2 (127.11.102.62:44777), 0a9e7156fd5c4d399486dc5932b407b2 (127.11.102.61:39977)
I20260620 06:07:16.355793 19086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:07:16.355565 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.102.62:44777 (local address 127.0.0.1:55220)
0620 06:07:16.355671 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0620 06:07:16.355728 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.102.62:44777: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
I20260620 06:07:16.356032 18381 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ec929d712e20476684cecae76f307d8e" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "0a9e7156fd5c4d399486dc5932b407b2"
I20260620 06:07:16.356051 18569 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "0a9e7156fd5c4d399486dc5932b407b2" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "ec929d712e20476684cecae76f307d8e"
I20260620 06:07:16.356176 19084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:07:16.355815 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.102.62:44777 (local address 127.0.0.1:55230)
0620 06:07:16.356038 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0620 06:07:16.356089 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.102.62:44777: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":164}
I20260620 06:07:16.356218 18569 raft_consensus.cc:2379] T 00000000000000000000000000000000 P ec929d712e20476684cecae76f307d8e [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 0a9e7156fd5c4d399486dc5932b407b2 in current term 2: Already voted for candidate ec929d712e20476684cecae76f307d8e in this term.
I20260620 06:07:16.356192 18381 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 0a9e7156fd5c4d399486dc5932b407b2 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate ec929d712e20476684cecae76f307d8e in current term 2: Already voted for candidate 0a9e7156fd5c4d399486dc5932b407b2 in this term.
I20260620 06:07:16.356711 19086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:07:16.356445 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.102.62:44777 (local address 127.0.0.1:55232)
0620 06:07:16.356605 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0620 06:07:16.356655 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.102.62:44777: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81}
W20260620 06:07:16.356802 18361 leader_election.cc:334] T 00000000000000000000000000000000 P 0a9e7156fd5c4d399486dc5932b407b2 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer d7d4500dc38a48a1858293d72d1fcfc2 (127.11.102.62:44777): Network error: Client connection negotiation failed: client connection to 127.11.102.62:44777: connect: Connection refused (error 111)
I20260620 06:07:16.356894 18361 leader_election.cc:302] T 00000000000000000000000000000000 P 0a9e7156fd5c4d399486dc5932b407b2 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a9e7156fd5c4d399486dc5932b407b2; no voters: d7d4500dc38a48a1858293d72d1fcfc2, ec929d712e20476684cecae76f307d8e
I20260620 06:07:16.357031 19084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:07:16.356784 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.102.62:44777 (local address 127.0.0.1:55238)
0620 06:07:16.356926 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0620 06:07:16.356975 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.102.62:44777: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
I20260620 06:07:16.357134 19080 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 0a9e7156fd5c4d399486dc5932b407b2 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
W20260620 06:07:16.357172 18544 leader_election.cc:334] T 00000000000000000000000000000000 P ec929d712e20476684cecae76f307d8e [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer d7d4500dc38a48a1858293d72d1fcfc2 (127.11.102.62:44777): Network error: Client connection negotiation failed: client connection to 127.11.102.62:44777: connect: Connection refused (error 111)
I20260620 06:07:16.357251 18544 leader_election.cc:302] T 00000000000000000000000000000000 P ec929d712e20476684cecae76f307d8e [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ec929d712e20476684cecae76f307d8e; no voters: 0a9e7156fd5c4d399486dc5932b407b2, d7d4500dc38a48a1858293d72d1fcfc2
I20260620 06:07:16.357383 19081 raft_consensus.cc:2726] T 00000000000000000000000000000000 P ec929d712e20476684cecae76f307d8e [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260620 06:07:16.837759 19090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:07:16.837251 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.102.62:44777 (local address 127.11.102.3:44123)
0620 06:07:16.837562 (+ 311us) negotiation.cc:107] Waiting for socket to connect
0620 06:07:16.837649 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.102.62:44777: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":131,"threads_started":1}
I20260620 06:07:16.841497 19091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:07:16.841057 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.102.62:44777 (local address 127.11.102.1:42691)
0620 06:07:16.841315 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0620 06:07:16.841370 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.102.62:44777: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":75,"threads_started":1}
I20260620 06:07:16.842996 19092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:07:16.842658 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.102.62:44777 (local address 127.11.102.2:49193)
0620 06:07:16.842873 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0620 06:07:16.842924 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.102.62:44777: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":153,"thread_start_us":77,"threads_started":1}
I20260620 06:07:17.838991 19093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:07:17.838499 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.102.62:44777 (local address 127.11.102.3:54565)
0620 06:07:17.838793 (+ 294us) negotiation.cc:107] Waiting for socket to connect
0620 06:07:17.838869 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.102.62:44777: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":216,"thread_start_us":111,"threads_started":1}
I20260620 06:07:17.842494 19094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:07:17.842112 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.102.62:44777 (local address 127.11.102.1:40159)
0620 06:07:17.842326 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0620 06:07:17.842391 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.102.62:44777: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":87,"threads_started":1}
I20260620 06:07:17.843880 19095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:07:17.843528 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.102.62:44777 (local address 127.11.102.2:32883)
0620 06:07:17.843768 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0620 06:07:17.843813 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.102.62:44777: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":164,"thread_start_us":51,"threads_started":1}
I20260620 06:07:17.948827 19096 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0a9e7156fd5c4d399486dc5932b407b2 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 06:07:17.949024 19096 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0a9e7156fd5c4d399486dc5932b407b2 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "d7d4500dc38a48a1858293d72d1fcfc2" member_type: VOTER last_known_addr { host: "127.11.102.62" port: 44777 } } peers { permanent_uuid: "0a9e7156fd5c4d399486dc5932b407b2" member_type: VOTER last_known_addr { host: "127.11.102.61" port: 39977 } attrs { promote: false } } peers { permanent_uuid: "ec929d712e20476684cecae76f307d8e" member_type: VOTER last_known_addr { host: "127.11.102.60" port: 41691 } attrs { promote: false } }
I20260620 06:07:17.949532 19096 leader_election.cc:288] T 00000000000000000000000000000000 P 0a9e7156fd5c4d399486dc5932b407b2 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d7d4500dc38a48a1858293d72d1fcfc2 (127.11.102.62:44777), ec929d712e20476684cecae76f307d8e (127.11.102.60:41691)
I20260620 06:07:17.950157 19097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:07:17.949769 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.102.62:44777 (local address 127.0.0.1:55462)
0620 06:07:17.949980 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0620 06:07:17.950055 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.102.62:44777: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":73,"threads_started":1}
I20260620 06:07:17.950379 18569 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "0a9e7156fd5c4d399486dc5932b407b2" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "ec929d712e20476684cecae76f307d8e" is_pre_election: true
I20260620 06:07:17.950553 18569 raft_consensus.cc:2454] T 00000000000000000000000000000000 P ec929d712e20476684cecae76f307d8e [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a9e7156fd5c4d399486dc5932b407b2 in term 2.
I20260620 06:07:17.950896 18361 leader_election.cc:302] T 00000000000000000000000000000000 P 0a9e7156fd5c4d399486dc5932b407b2 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a9e7156fd5c4d399486dc5932b407b2, ec929d712e20476684cecae76f307d8e; no voters:
I20260620 06:07:17.951022 19097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:07:17.950755 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.102.62:44777 (local address 127.0.0.1:55478)
0620 06:07:17.950879 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0620 06:07:17.950937 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.102.62:44777: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260620 06:07:17.951062 19096 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0a9e7156fd5c4d399486dc5932b407b2 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260620 06:07:17.951109 19096 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0a9e7156fd5c4d399486dc5932b407b2 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 06:07:17.951172 19096 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0a9e7156fd5c4d399486dc5932b407b2 [term 2 FOLLOWER]: Advancing to term 3
W20260620 06:07:17.951181 18361 leader_election.cc:334] T 00000000000000000000000000000000 P 0a9e7156fd5c4d399486dc5932b407b2 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer d7d4500dc38a48a1858293d72d1fcfc2 (127.11.102.62:44777): Network error: Client connection negotiation failed: client connection to 127.11.102.62:44777: connect: Connection refused (error 111)
I20260620 06:07:17.952260 19096 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0a9e7156fd5c4d399486dc5932b407b2 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "d7d4500dc38a48a1858293d72d1fcfc2" member_type: VOTER last_known_addr { host: "127.11.102.62" port: 44777 } } peers { permanent_uuid: "0a9e7156fd5c4d399486dc5932b407b2" member_type: VOTER last_known_addr { host: "127.11.102.61" port: 39977 } attrs { promote: false } } peers { permanent_uuid: "ec929d712e20476684cecae76f307d8e" member_type: VOTER last_known_addr { host: "127.11.102.60" port: 41691 } attrs { promote: false } }
I20260620 06:07:17.952561 19096 leader_election.cc:288] T 00000000000000000000000000000000 P 0a9e7156fd5c4d399486dc5932b407b2 [CANDIDATE]: Term 3 election: Requested vote from peers d7d4500dc38a48a1858293d72d1fcfc2 (127.11.102.62:44777), ec929d712e20476684cecae76f307d8e (127.11.102.60:41691)
I20260620 06:07:17.952988 19097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:07:17.952758 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.102.62:44777 (local address 127.0.0.1:55486)
0620 06:07:17.952845 (+ 87us) negotiation.cc:107] Waiting for socket to connect
0620 06:07:17.952935 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.102.62:44777: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
I20260620 06:07:17.953132 18569 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "0a9e7156fd5c4d399486dc5932b407b2" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "ec929d712e20476684cecae76f307d8e"
I20260620 06:07:17.953258 18569 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ec929d712e20476684cecae76f307d8e [term 2 FOLLOWER]: Advancing to term 3
I20260620 06:07:17.953593 19097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:07:17.953395 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.102.62:44777 (local address 127.0.0.1:55488)
0620 06:07:17.953493 (+ 98us) negotiation.cc:107] Waiting for socket to connect
0620 06:07:17.953537 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.102.62:44777: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260620 06:07:17.953689 18361 leader_election.cc:334] T 00000000000000000000000000000000 P 0a9e7156fd5c4d399486dc5932b407b2 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer d7d4500dc38a48a1858293d72d1fcfc2 (127.11.102.62:44777): Network error: Client connection negotiation failed: client connection to 127.11.102.62:44777: connect: Connection refused (error 111)
I20260620 06:07:17.954276 18569 raft_consensus.cc:2454] T 00000000000000000000000000000000 P ec929d712e20476684cecae76f307d8e [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a9e7156fd5c4d399486dc5932b407b2 in term 3.
I20260620 06:07:17.954596 18361 leader_election.cc:302] T 00000000000000000000000000000000 P 0a9e7156fd5c4d399486dc5932b407b2 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a9e7156fd5c4d399486dc5932b407b2, ec929d712e20476684cecae76f307d8e; no voters: d7d4500dc38a48a1858293d72d1fcfc2
I20260620 06:07:17.954764 19096 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0a9e7156fd5c4d399486dc5932b407b2 [term 3 FOLLOWER]: Leader election won for term 3
I20260620 06:07:17.954952 19096 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0a9e7156fd5c4d399486dc5932b407b2 [term 3 LEADER]: Becoming Leader. State: Replica: 0a9e7156fd5c4d399486dc5932b407b2, State: Running, Role: LEADER
I20260620 06:07:17.955200 19096 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0a9e7156fd5c4d399486dc5932b407b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "d7d4500dc38a48a1858293d72d1fcfc2" member_type: VOTER last_known_addr { host: "127.11.102.62" port: 44777 } } peers { permanent_uuid: "0a9e7156fd5c4d399486dc5932b407b2" member_type: VOTER last_known_addr { host: "127.11.102.61" port: 39977 } attrs { promote: false } } peers { permanent_uuid: "ec929d712e20476684cecae76f307d8e" member_type: VOTER last_known_addr { host: "127.11.102.60" port: 41691 } attrs { promote: false } }
I20260620 06:07:17.956280 19099 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0a9e7156fd5c4d399486dc5932b407b2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0a9e7156fd5c4d399486dc5932b407b2. Latest consensus state: current_term: 3 leader_uuid: "0a9e7156fd5c4d399486dc5932b407b2" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "d7d4500dc38a48a1858293d72d1fcfc2" member_type: VOTER last_known_addr { host: "127.11.102.62" port: 44777 } } peers { permanent_uuid: "0a9e7156fd5c4d399486dc5932b407b2" member_type: VOTER last_known_addr { host: "127.11.102.61" port: 39977 } attrs { promote: false } } peers { permanent_uuid: "ec929d712e20476684cecae76f307d8e" member_type: VOTER last_known_addr { host: "127.11.102.60" port: 41691 } attrs { promote: false } } }
I20260620 06:07:17.956403 19099 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0a9e7156fd5c4d399486dc5932b407b2 [sys.catalog]: This master's current role is: LEADER
I20260620 06:07:17.956734 19101 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 06:07:17.959018 19101 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 06:07:17.959407 19101 catalog_manager.cc:1259] Loaded cluster ID: bcc977c4637644dab45ae102d36c03ef
I20260620 06:07:17.959476 19101 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 06:07:17.960070 19101 catalog_manager.cc:1504] Loading token signing keys...
I20260620 06:07:17.960443 19101 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 0a9e7156fd5c4d399486dc5932b407b2: Loaded TSK: 0
I20260620 06:07:17.961097 19101 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 06:07:18.008038 18368 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52524:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 06:07:18.011029 18368 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 06:07:18.013931 19097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:07:18.013715 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.102.62:44777 (local address 127.0.0.1:55496)
0620 06:07:18.013820 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0620 06:07:18.013869 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.102.62:44777: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20260620 06:07:18.014196 18569 raft_consensus.cc:1261] T 00000000000000000000000000000000 P ec929d712e20476684cecae76f307d8e [term 3 FOLLOWER]: Refusing update from remote peer 0a9e7156fd5c4d399486dc5932b407b2: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 3 index: 14. (index mismatch)
I20260620 06:07:18.014634 19097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:07:18.014388 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.102.62:44777 (local address 127.0.0.1:55510)
0620 06:07:18.014507 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0620 06:07:18.014566 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.102.62:44777: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260620 06:07:18.014662 19096 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 0a9e7156fd5c4d399486dc5932b407b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec929d712e20476684cecae76f307d8e" member_type: VOTER last_known_addr { host: "127.11.102.60" port: 41691 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
W20260620 06:07:18.014868 18361 consensus_peers.cc:487] T 00000000000000000000000000000000 P 0a9e7156fd5c4d399486dc5932b407b2 -> Peer d7d4500dc38a48a1858293d72d1fcfc2 (127.11.102.62:44777): Couldn't send request to peer d7d4500dc38a48a1858293d72d1fcfc2. Status: Network error: Client connection negotiation failed: client connection to 127.11.102.62:44777: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260620 06:07:18.016180 19103 sys_catalog.cc:455] T 00000000000000000000000000000000 P ec929d712e20476684cecae76f307d8e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0a9e7156fd5c4d399486dc5932b407b2. Latest consensus state: current_term: 3 leader_uuid: "0a9e7156fd5c4d399486dc5932b407b2" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "d7d4500dc38a48a1858293d72d1fcfc2" member_type: VOTER last_known_addr { host: "127.11.102.62" port: 44777 } } peers { permanent_uuid: "0a9e7156fd5c4d399486dc5932b407b2" member_type: VOTER last_known_addr { host: "127.11.102.61" port: 39977 } attrs { promote: false } } peers { permanent_uuid: "ec929d712e20476684cecae76f307d8e" member_type: VOTER last_known_addr { host: "127.11.102.60" port: 41691 } attrs { promote: false } } }
I20260620 06:07:18.016307 19103 sys_catalog.cc:458] T 00000000000000000000000000000000 P ec929d712e20476684cecae76f307d8e [sys.catalog]: This master's current role is: FOLLOWER
I20260620 06:07:18.017385 19096 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0a9e7156fd5c4d399486dc5932b407b2 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 3 leader_uuid: "0a9e7156fd5c4d399486dc5932b407b2" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "d7d4500dc38a48a1858293d72d1fcfc2" member_type: VOTER last_known_addr { host: "127.11.102.62" port: 44777 } } peers { permanent_uuid: "0a9e7156fd5c4d399486dc5932b407b2" member_type: VOTER last_known_addr { host: "127.11.102.61" port: 39977 } attrs { promote: false } } peers { permanent_uuid: "ec929d712e20476684cecae76f307d8e" member_type: VOTER last_known_addr { host: "127.11.102.60" port: 41691 } attrs { promote: false } } }
I20260620 06:07:18.017504 19096 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0a9e7156fd5c4d399486dc5932b407b2 [sys.catalog]: This master's current role is: LEADER
I20260620 06:07:18.017664 19103 sys_catalog.cc:455] T 00000000000000000000000000000000 P ec929d712e20476684cecae76f307d8e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 3 leader_uuid: "0a9e7156fd5c4d399486dc5932b407b2" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "d7d4500dc38a48a1858293d72d1fcfc2" member_type: VOTER last_known_addr { host: "127.11.102.62" port: 44777 } } peers { permanent_uuid: "0a9e7156fd5c4d399486dc5932b407b2" member_type: VOTER last_known_addr { host: "127.11.102.61" port: 39977 } attrs { promote: false } } peers { permanent_uuid: "ec929d712e20476684cecae76f307d8e" member_type: VOTER last_known_addr { host: "127.11.102.60" port: 41691 } attrs { promote: false } } }
I20260620 06:07:18.017786 19103 sys_catalog.cc:458] T 00000000000000000000000000000000 P ec929d712e20476684cecae76f307d8e [sys.catalog]: This master's current role is: FOLLOWER
I20260620 06:07:18.027894 19111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:07:18.023367 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52238 (local address 127.11.102.1:37681)
0620 06:07:18.023669 (+ 302us) server_negotiation.cc:206] Beginning negotiation
0620 06:07:18.023674 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 06:07:18.023830 (+ 156us) server_negotiation.cc:407] Connection header received
0620 06:07:18.023953 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:07:18.023956 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:07:18.024014 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:07:18.024101 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 06:07:18.024412 (+ 311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:07:18.025200 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:07:18.026850 (+ 1650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:07:18.027264 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:07:18.027384 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:07:18.027454 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0620 06:07:18.027685 (+ 231us) server_negotiation.cc:299] Negotiation successful
0620 06:07:18.027757 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":192,"thread_start_us":107,"threads_started":1}
I20260620 06:07:18.028323 19112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:07:18.023184 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.102.1:37681 (local address 127.0.0.1:52238)
0620 06:07:18.023737 (+ 553us) negotiation.cc:107] Waiting for socket to connect
0620 06:07:18.023749 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 06:07:18.023861 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:07:18.024216 (+ 355us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:07:18.024218 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:07:18.024232 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 06:07:18.024308 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:07:18.024312 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:07:18.025336 (+ 1024us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:07:18.025341 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:07:18.026659 (+ 1318us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:07:18.026669 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:07:18.027380 (+ 711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:07:18.027384 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:07:18.027506 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:07:18.027528 (+ 22us) client_negotiation.cc:769] Sending connection context
0620 06:07:18.028093 (+ 565us) client_negotiation.cc:240] Negotiation successful
0620 06:07:18.028170 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":499,"mutex_wait_us":64,"thread_start_us":79,"threads_started":1}
I20260620 06:07:18.028750 19113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:07:18.023583 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59314 (local address 127.11.102.2:33643)
0620 06:07:18.023853 (+ 270us) server_negotiation.cc:206] Beginning negotiation
0620 06:07:18.023857 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 06:07:18.024191 (+ 334us) server_negotiation.cc:407] Connection header received
0620 06:07:18.024251 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:07:18.024255 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:07:18.024302 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:07:18.024376 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 06:07:18.024707 (+ 331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:07:18.025214 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:07:18.027535 (+ 2321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:07:18.028063 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:07:18.028162 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:07:18.028218 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0620 06:07:18.028410 (+ 192us) server_negotiation.cc:299] Negotiation successful
0620 06:07:18.028611 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":175,"thread_start_us":84,"threads_started":1}
I20260620 06:07:18.028910 18672 tablet_service.cc:1467] Processing CreateTablet for tablet 1b84faf2ec3c4038bbfee0fac00de79f (DEFAULT_TABLE table=test-table [id=e6701a3312a642799248405742bdb48e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 06:07:18.028968 19097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:07:18.022927 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.102.3:36875 (local address 127.0.0.1:58698)
0620 06:07:18.023031 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0620 06:07:18.023045 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 06:07:18.023132 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:07:18.024603 (+ 1471us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:07:18.024606 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:07:18.024623 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 06:07:18.024698 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:07:18.024704 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:07:18.025553 (+ 849us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:07:18.025555 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:07:18.026664 (+ 1109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:07:18.026669 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:07:18.028575 (+ 1906us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:07:18.028579 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:07:18.028668 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:07:18.028693 (+ 25us) client_negotiation.cc:769] Sending connection context
0620 06:07:18.028785 (+ 92us) client_negotiation.cc:240] Negotiation successful
0620 06:07:18.028853 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":42,"mutex_wait_us":21}
I20260620 06:07:18.029102 19109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:07:18.023070 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58698 (local address 127.11.102.3:36875)
0620 06:07:18.024377 (+ 1307us) server_negotiation.cc:206] Beginning negotiation
0620 06:07:18.024381 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 06:07:18.024393 (+ 12us) server_negotiation.cc:407] Connection header received
0620 06:07:18.024438 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:07:18.024440 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:07:18.024487 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:07:18.024567 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 06:07:18.024827 (+ 260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:07:18.025487 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:07:18.027506 (+ 2019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:07:18.028306 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:07:18.028570 (+ 264us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:07:18.028629 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0620 06:07:18.028811 (+ 182us) server_negotiation.cc:299] Negotiation successful
0620 06:07:18.028962 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1210,"thread_start_us":70,"threads_started":1}
I20260620 06:07:18.029227 18672 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1b84faf2ec3c4038bbfee0fac00de79f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 06:07:18.029567 18946 tablet_service.cc:1467] Processing CreateTablet for tablet 1b84faf2ec3c4038bbfee0fac00de79f (DEFAULT_TABLE table=test-table [id=e6701a3312a642799248405742bdb48e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 06:07:18.029817 18946 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1b84faf2ec3c4038bbfee0fac00de79f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 06:07:18.031809 19110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:07:18.023458 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.102.2:33643 (local address 127.0.0.1:59314)
0620 06:07:18.024036 (+ 578us) negotiation.cc:107] Waiting for socket to connect
0620 06:07:18.024047 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 06:07:18.024159 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:07:18.024482 (+ 323us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:07:18.024485 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:07:18.024501 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 06:07:18.024615 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:07:18.024621 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:07:18.025348 (+ 727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:07:18.025350 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:07:18.027417 (+ 2067us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:07:18.027424 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:07:18.028162 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:07:18.028166 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:07:18.028260 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:07:18.028309 (+ 49us) client_negotiation.cc:769] Sending connection context
0620 06:07:18.028396 (+ 87us) client_negotiation.cc:240] Negotiation successful
0620 06:07:18.028463 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":526,"mutex_wait_us":1266,"thread_start_us":74,"threads_started":1}
I20260620 06:07:18.031817 19114 tablet_bootstrap.cc:492] T 1b84faf2ec3c4038bbfee0fac00de79f P fb0783328ad3442fba39f260adf25308: Bootstrap starting.
I20260620 06:07:18.032361 19115 tablet_bootstrap.cc:492] T 1b84faf2ec3c4038bbfee0fac00de79f P d4b3fbe7b8664c968f75cc26b72410d7: Bootstrap starting.
I20260620 06:07:18.032491 18809 tablet_service.cc:1467] Processing CreateTablet for tablet 1b84faf2ec3c4038bbfee0fac00de79f (DEFAULT_TABLE table=test-table [id=e6701a3312a642799248405742bdb48e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 06:07:18.032698 19114 tablet_bootstrap.cc:654] T 1b84faf2ec3c4038bbfee0fac00de79f P fb0783328ad3442fba39f260adf25308: Neither blocks nor log segments found. Creating new log.
I20260620 06:07:18.032788 18809 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1b84faf2ec3c4038bbfee0fac00de79f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 06:07:18.033334 19115 tablet_bootstrap.cc:654] T 1b84faf2ec3c4038bbfee0fac00de79f P d4b3fbe7b8664c968f75cc26b72410d7: Neither blocks nor log segments found. Creating new log.
I20260620 06:07:18.034456 19115 tablet_bootstrap.cc:492] T 1b84faf2ec3c4038bbfee0fac00de79f P d4b3fbe7b8664c968f75cc26b72410d7: No bootstrap required, opened a new log
I20260620 06:07:18.034569 19115 ts_tablet_manager.cc:1397] T 1b84faf2ec3c4038bbfee0fac00de79f P d4b3fbe7b8664c968f75cc26b72410d7: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260620 06:07:18.034971 19118 tablet_bootstrap.cc:492] T 1b84faf2ec3c4038bbfee0fac00de79f P f569ba0a0ea74d078dabe54eb42b9e21: Bootstrap starting.
I20260620 06:07:18.034960 19115 raft_consensus.cc:348] T 1b84faf2ec3c4038bbfee0fac00de79f P d4b3fbe7b8664c968f75cc26b72410d7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4b3fbe7b8664c968f75cc26b72410d7" member_type: VOTER last_known_addr { host: "127.11.102.3" port: 36875 } } peers { permanent_uuid: "fb0783328ad3442fba39f260adf25308" member_type: VOTER last_known_addr { host: "127.11.102.1" port: 37681 } } peers { permanent_uuid: "f569ba0a0ea74d078dabe54eb42b9e21" member_type: VOTER last_known_addr { host: "127.11.102.2" port: 33643 } }
I20260620 06:07:18.035099 19115 raft_consensus.cc:374] T 1b84faf2ec3c4038bbfee0fac00de79f P d4b3fbe7b8664c968f75cc26b72410d7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 06:07:18.035140 19115 raft_consensus.cc:729] T 1b84faf2ec3c4038bbfee0fac00de79f P d4b3fbe7b8664c968f75cc26b72410d7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d4b3fbe7b8664c968f75cc26b72410d7, State: Initialized, Role: FOLLOWER
I20260620 06:07:18.035264 19115 consensus_queue.cc:260] T 1b84faf2ec3c4038bbfee0fac00de79f P d4b3fbe7b8664c968f75cc26b72410d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4b3fbe7b8664c968f75cc26b72410d7" member_type: VOTER last_known_addr { host: "127.11.102.3" port: 36875 } } peers { permanent_uuid: "fb0783328ad3442fba39f260adf25308" member_type: VOTER last_known_addr { host: "127.11.102.1" port: 37681 } } peers { permanent_uuid: "f569ba0a0ea74d078dabe54eb42b9e21" member_type: VOTER last_known_addr { host: "127.11.102.2" port: 33643 } }
I20260620 06:07:18.035869 19115 ts_tablet_manager.cc:1428] T 1b84faf2ec3c4038bbfee0fac00de79f P d4b3fbe7b8664c968f75cc26b72410d7: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20260620 06:07:18.035974 19118 tablet_bootstrap.cc:654] T 1b84faf2ec3c4038bbfee0fac00de79f P f569ba0a0ea74d078dabe54eb42b9e21: Neither blocks nor log segments found. Creating new log.
I20260620 06:07:18.036564 19093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:07:18.036323 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.102.62:44777 (local address 127.11.102.3:52123)
0620 06:07:18.036426 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0620 06:07:18.036482 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.102.62:44777: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
I20260620 06:07:18.037258 19093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:07:18.037050 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.102.62:44777 (local address 127.11.102.3:43139)
0620 06:07:18.037155 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0620 06:07:18.037203 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.102.62:44777: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260620 06:07:18.038115 19114 tablet_bootstrap.cc:492] T 1b84faf2ec3c4038bbfee0fac00de79f P fb0783328ad3442fba39f260adf25308: No bootstrap required, opened a new log
I20260620 06:07:18.038316 19114 ts_tablet_manager.cc:1397] T 1b84faf2ec3c4038bbfee0fac00de79f P fb0783328ad3442fba39f260adf25308: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20260620 06:07:18.038552 19118 tablet_bootstrap.cc:492] T 1b84faf2ec3c4038bbfee0fac00de79f P f569ba0a0ea74d078dabe54eb42b9e21: No bootstrap required, opened a new log
I20260620 06:07:18.038645 19118 ts_tablet_manager.cc:1397] T 1b84faf2ec3c4038bbfee0fac00de79f P f569ba0a0ea74d078dabe54eb42b9e21: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260620 06:07:18.038782 19114 raft_consensus.cc:348] T 1b84faf2ec3c4038bbfee0fac00de79f P fb0783328ad3442fba39f260adf25308 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4b3fbe7b8664c968f75cc26b72410d7" member_type: VOTER last_known_addr { host: "127.11.102.3" port: 36875 } } peers { permanent_uuid: "fb0783328ad3442fba39f260adf25308" member_type: VOTER last_known_addr { host: "127.11.102.1" port: 37681 } } peers { permanent_uuid: "f569ba0a0ea74d078dabe54eb42b9e21" member_type: VOTER last_known_addr { host: "127.11.102.2" port: 33643 } }
I20260620 06:07:18.038882 19114 raft_consensus.cc:374] T 1b84faf2ec3c4038bbfee0fac00de79f P fb0783328ad3442fba39f260adf25308 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 06:07:18.038924 19114 raft_consensus.cc:729] T 1b84faf2ec3c4038bbfee0fac00de79f P fb0783328ad3442fba39f260adf25308 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fb0783328ad3442fba39f260adf25308, State: Initialized, Role: FOLLOWER
I20260620 06:07:18.039000 19118 raft_consensus.cc:348] T 1b84faf2ec3c4038bbfee0fac00de79f P f569ba0a0ea74d078dabe54eb42b9e21 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4b3fbe7b8664c968f75cc26b72410d7" member_type: VOTER last_known_addr { host: "127.11.102.3" port: 36875 } } peers { permanent_uuid: "fb0783328ad3442fba39f260adf25308" member_type: VOTER last_known_addr { host: "127.11.102.1" port: 37681 } } peers { permanent_uuid: "f569ba0a0ea74d078dabe54eb42b9e21" member_type: VOTER last_known_addr { host: "127.11.102.2" port: 33643 } }
I20260620 06:07:18.039105 19118 raft_consensus.cc:374] T 1b84faf2ec3c4038bbfee0fac00de79f P f569ba0a0ea74d078dabe54eb42b9e21 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 06:07:18.039062 19114 consensus_queue.cc:260] T 1b84faf2ec3c4038bbfee0fac00de79f P fb0783328ad3442fba39f260adf25308 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4b3fbe7b8664c968f75cc26b72410d7" member_type: VOTER last_known_addr { host: "127.11.102.3" port: 36875 } } peers { permanent_uuid: "fb0783328ad3442fba39f260adf25308" member_type: VOTER last_known_addr { host: "127.11.102.1" port: 37681 } } peers { permanent_uuid: "f569ba0a0ea74d078dabe54eb42b9e21" member_type: VOTER last_known_addr { host: "127.11.102.2" port: 33643 } }
I20260620 06:07:18.039150 19118 raft_consensus.cc:729] T 1b84faf2ec3c4038bbfee0fac00de79f P f569ba0a0ea74d078dabe54eb42b9e21 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f569ba0a0ea74d078dabe54eb42b9e21, State: Initialized, Role: FOLLOWER
I20260620 06:07:18.039269 19118 consensus_queue.cc:260] T 1b84faf2ec3c4038bbfee0fac00de79f P f569ba0a0ea74d078dabe54eb42b9e21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4b3fbe7b8664c968f75cc26b72410d7" member_type: VOTER last_known_addr { host: "127.11.102.3" port: 36875 } } peers { permanent_uuid: "fb0783328ad3442fba39f260adf25308" member_type: VOTER last_known_addr { host: "127.11.102.1" port: 37681 } } peers { permanent_uuid: "f569ba0a0ea74d078dabe54eb42b9e21" member_type: VOTER last_known_addr { host: "127.11.102.2" port: 33643 } }
I20260620 06:07:18.039443 19114 ts_tablet_manager.cc:1428] T 1b84faf2ec3c4038bbfee0fac00de79f P fb0783328ad3442fba39f260adf25308: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260620 06:07:18.039669 19118 ts_tablet_manager.cc:1428] T 1b84faf2ec3c4038bbfee0fac00de79f P f569ba0a0ea74d078dabe54eb42b9e21: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260620 06:07:18.040031 19094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:07:18.039699 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.102.62:44777 (local address 127.11.102.1:35785)
0620 06:07:18.039914 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0620 06:07:18.039967 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.102.62:44777: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":151}
I20260620 06:07:18.040796 19095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:07:18.040009 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.102.62:44777 (local address 127.11.102.2:40023)
0620 06:07:18.040681 (+ 672us) negotiation.cc:107] Waiting for socket to connect
0620 06:07:18.040732 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.102.62:44777: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":606}
I20260620 06:07:18.041134 18877 heartbeater.cc:503] Master 127.11.102.61:39977 was elected leader, sending a full tablet report...
I20260620 06:07:18.042941 19094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:07:18.042600 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.102.62:44777 (local address 127.11.102.1:52001)
0620 06:07:18.042825 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0620 06:07:18.042875 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.102.62:44777: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":167}
I20260620 06:07:18.043831 19095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:07:18.041590 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.102.62:44777 (local address 127.11.102.2:38657)
0620 06:07:18.043716 (+ 2126us) negotiation.cc:107] Waiting for socket to connect
0620 06:07:18.043767 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.102.62:44777: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":2045}
I20260620 06:07:18.043987 18740 heartbeater.cc:503] Master 127.11.102.61:39977 was elected leader, sending a full tablet report...
I20260620 06:07:18.134795 19122 raft_consensus.cc:482] T 1b84faf2ec3c4038bbfee0fac00de79f P f569ba0a0ea74d078dabe54eb42b9e21 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 06:07:18.134938 19122 raft_consensus.cc:504] T 1b84faf2ec3c4038bbfee0fac00de79f P f569ba0a0ea74d078dabe54eb42b9e21 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4b3fbe7b8664c968f75cc26b72410d7" member_type: VOTER last_known_addr { host: "127.11.102.3" port: 36875 } } peers { permanent_uuid: "fb0783328ad3442fba39f260adf25308" member_type: VOTER last_known_addr { host: "127.11.102.1" port: 37681 } } peers { permanent_uuid: "f569ba0a0ea74d078dabe54eb42b9e21" member_type: VOTER last_known_addr { host: "127.11.102.2" port: 33643 } }
I20260620 06:07:18.135327 19122 leader_election.cc:288] T 1b84faf2ec3c4038bbfee0fac00de79f P f569ba0a0ea74d078dabe54eb42b9e21 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d4b3fbe7b8664c968f75cc26b72410d7 (127.11.102.3:36875), fb0783328ad3442fba39f260adf25308 (127.11.102.1:37681)
I20260620 06:07:18.135782 18692 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b84faf2ec3c4038bbfee0fac00de79f" candidate_uuid: "f569ba0a0ea74d078dabe54eb42b9e21" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fb0783328ad3442fba39f260adf25308" is_pre_election: true
I20260620 06:07:18.135848 18966 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b84faf2ec3c4038bbfee0fac00de79f" candidate_uuid: "f569ba0a0ea74d078dabe54eb42b9e21" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4b3fbe7b8664c968f75cc26b72410d7" is_pre_election: true
I20260620 06:07:18.135972 18692 raft_consensus.cc:2454] T 1b84faf2ec3c4038bbfee0fac00de79f P fb0783328ad3442fba39f260adf25308 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f569ba0a0ea74d078dabe54eb42b9e21 in term 0.
I20260620 06:07:18.136018 18966 raft_consensus.cc:2454] T 1b84faf2ec3c4038bbfee0fac00de79f P d4b3fbe7b8664c968f75cc26b72410d7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f569ba0a0ea74d078dabe54eb42b9e21 in term 0.
I20260620 06:07:18.136340 18763 leader_election.cc:302] T 1b84faf2ec3c4038bbfee0fac00de79f P f569ba0a0ea74d078dabe54eb42b9e21 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d4b3fbe7b8664c968f75cc26b72410d7, f569ba0a0ea74d078dabe54eb42b9e21; no voters:
I20260620 06:07:18.136554 19122 raft_consensus.cc:2781] T 1b84faf2ec3c4038bbfee0fac00de79f P f569ba0a0ea74d078dabe54eb42b9e21 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 06:07:18.136615 19122 raft_consensus.cc:482] T 1b84faf2ec3c4038bbfee0fac00de79f P f569ba0a0ea74d078dabe54eb42b9e21 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 06:07:18.136667 19122 raft_consensus.cc:3037] T 1b84faf2ec3c4038bbfee0fac00de79f P f569ba0a0ea74d078dabe54eb42b9e21 [term 0 FOLLOWER]: Advancing to term 1
I20260620 06:07:18.137817 19122 raft_consensus.cc:504] T 1b84faf2ec3c4038bbfee0fac00de79f P f569ba0a0ea74d078dabe54eb42b9e21 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4b3fbe7b8664c968f75cc26b72410d7" member_type: VOTER last_known_addr { host: "127.11.102.3" port: 36875 } } peers { permanent_uuid: "fb0783328ad3442fba39f260adf25308" member_type: VOTER last_known_addr { host: "127.11.102.1" port: 37681 } } peers { permanent_uuid: "f569ba0a0ea74d078dabe54eb42b9e21" member_type: VOTER last_known_addr { host: "127.11.102.2" port: 33643 } }
I20260620 06:07:18.138190 19122 leader_election.cc:288] T 1b84faf2ec3c4038bbfee0fac00de79f P f569ba0a0ea74d078dabe54eb42b9e21 [CANDIDATE]: Term 1 election: Requested vote from peers d4b3fbe7b8664c968f75cc26b72410d7 (127.11.102.3:36875), fb0783328ad3442fba39f260adf25308 (127.11.102.1:37681)
I20260620 06:07:18.138545 18966 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b84faf2ec3c4038bbfee0fac00de79f" candidate_uuid: "f569ba0a0ea74d078dabe54eb42b9e21" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4b3fbe7b8664c968f75cc26b72410d7"
I20260620 06:07:18.138579 18692 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b84faf2ec3c4038bbfee0fac00de79f" candidate_uuid: "f569ba0a0ea74d078dabe54eb42b9e21" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fb0783328ad3442fba39f260adf25308"
I20260620 06:07:18.138664 18966 raft_consensus.cc:3037] T 1b84faf2ec3c4038bbfee0fac00de79f P d4b3fbe7b8664c968f75cc26b72410d7 [term 0 FOLLOWER]: Advancing to term 1
I20260620 06:07:18.138692 18692 raft_consensus.cc:3037] T 1b84faf2ec3c4038bbfee0fac00de79f P fb0783328ad3442fba39f260adf25308 [term 0 FOLLOWER]: Advancing to term 1
I20260620 06:07:18.139635 18692 raft_consensus.cc:2454] T 1b84faf2ec3c4038bbfee0fac00de79f P fb0783328ad3442fba39f260adf25308 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f569ba0a0ea74d078dabe54eb42b9e21 in term 1.
I20260620 06:07:18.139681 18966 raft_consensus.cc:2454] T 1b84faf2ec3c4038bbfee0fac00de79f P d4b3fbe7b8664c968f75cc26b72410d7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f569ba0a0ea74d078dabe54eb42b9e21 in term 1.
I20260620 06:07:18.139917 18764 leader_election.cc:302] T 1b84faf2ec3c4038bbfee0fac00de79f P f569ba0a0ea74d078dabe54eb42b9e21 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: f569ba0a0ea74d078dabe54eb42b9e21, fb0783328ad3442fba39f260adf25308; no voters:
I20260620 06:07:18.140123 19122 raft_consensus.cc:2781] T 1b84faf2ec3c4038bbfee0fac00de79f P f569ba0a0ea74d078dabe54eb42b9e21 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 06:07:18.140365 19122 raft_consensus.cc:686] T 1b84faf2ec3c4038bbfee0fac00de79f P f569ba0a0ea74d078dabe54eb42b9e21 [term 1 LEADER]: Becoming Leader. State: Replica: f569ba0a0ea74d078dabe54eb42b9e21, State: Running, Role: LEADER
I20260620 06:07:18.140518 19122 consensus_queue.cc:237] T 1b84faf2ec3c4038bbfee0fac00de79f P f569ba0a0ea74d078dabe54eb42b9e21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4b3fbe7b8664c968f75cc26b72410d7" member_type: VOTER last_known_addr { host: "127.11.102.3" port: 36875 } } peers { permanent_uuid: "fb0783328ad3442fba39f260adf25308" member_type: VOTER last_known_addr { host: "127.11.102.1" port: 37681 } } peers { permanent_uuid: "f569ba0a0ea74d078dabe54eb42b9e21" member_type: VOTER last_known_addr { host: "127.11.102.2" port: 33643 } }
I20260620 06:07:18.141098 19095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:07:18.140857 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.102.62:44777 (local address 127.11.102.2:42747)
0620 06:07:18.140954 (+ 97us) negotiation.cc:107] Waiting for socket to connect
0620 06:07:18.141023 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.102.62:44777: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
I20260620 06:07:18.142397 18368 catalog_manager.cc:5495] T 1b84faf2ec3c4038bbfee0fac00de79f P f569ba0a0ea74d078dabe54eb42b9e21 reported cstate change: term changed from 0 to 1, leader changed from <none> to f569ba0a0ea74d078dabe54eb42b9e21 (127.11.102.2). New cstate: current_term: 1 leader_uuid: "f569ba0a0ea74d078dabe54eb42b9e21" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4b3fbe7b8664c968f75cc26b72410d7" member_type: VOTER last_known_addr { host: "127.11.102.3" port: 36875 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fb0783328ad3442fba39f260adf25308" member_type: VOTER last_known_addr { host: "127.11.102.1" port: 37681 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f569ba0a0ea74d078dabe54eb42b9e21" member_type: VOTER last_known_addr { host: "127.11.102.2" port: 33643 } health_report { overall_health: HEALTHY } } }
I20260620 06:07:18.177685 19113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:07:18.174536 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59326 (local address 127.11.102.2:33643)
0620 06:07:18.174701 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0620 06:07:18.174706 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 06:07:18.174722 (+ 16us) server_negotiation.cc:407] Connection header received
0620 06:07:18.174782 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:07:18.174785 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:07:18.174837 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:07:18.174919 (+ 82us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 06:07:18.175195 (+ 276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:07:18.175802 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:07:18.176634 (+ 832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:07:18.176853 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:07:18.176966 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:07:18.177215 (+ 249us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 06:07:18.177281 (+ 66us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 06:07:18.177361 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0620 06:07:18.177480 (+ 119us) server_negotiation.cc:299] Negotiation successful
0620 06:07:18.177533 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260620 06:07:18.179814 18692 raft_consensus.cc:1261] T 1b84faf2ec3c4038bbfee0fac00de79f P fb0783328ad3442fba39f260adf25308 [term 1 FOLLOWER]: Refusing update from remote peer f569ba0a0ea74d078dabe54eb42b9e21: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 06:07:18.179816 18966 raft_consensus.cc:1261] T 1b84faf2ec3c4038bbfee0fac00de79f P d4b3fbe7b8664c968f75cc26b72410d7 [term 1 FOLLOWER]: Refusing update from remote peer f569ba0a0ea74d078dabe54eb42b9e21: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 06:07:18.180294 19123 consensus_queue.cc:1035] T 1b84faf2ec3c4038bbfee0fac00de79f P f569ba0a0ea74d078dabe54eb42b9e21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fb0783328ad3442fba39f260adf25308" member_type: VOTER last_known_addr { host: "127.11.102.1" port: 37681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 06:07:18.180464 19122 consensus_queue.cc:1035] T 1b84faf2ec3c4038bbfee0fac00de79f P f569ba0a0ea74d078dabe54eb42b9e21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4b3fbe7b8664c968f75cc26b72410d7" member_type: VOTER last_known_addr { host: "127.11.102.3" port: 36875 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 06:07:18.181491 19014 heartbeater.cc:503] Master 127.11.102.61:39977 was elected leader, sending a full tablet report...
I20260620 06:07:18.182325 19094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:07:18.182023 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.102.62:44777 (local address 127.11.102.1:40431)
0620 06:07:18.182183 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0620 06:07:18.182246 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.102.62:44777: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94}
I20260620 06:07:18.182425 19093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:07:18.182103 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.102.62:44777 (local address 127.11.102.3:37197)
0620 06:07:18.182306 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0620 06:07:18.182357 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.102.62:44777: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":145}
I20260620 06:07:18.185092 19126 mvcc.cc:205] Tried to move back new op lower bound from 7298808373980979200 to 7298808373826449408. Current Snapshot: MvccSnapshot[applied={T|T < 7298808373980979200}]
I20260620 06:07:18.185813 19095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:07:18.185563 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.102.62:44777 (local address 127.11.102.2:39517)
0620 06:07:18.185668 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0620 06:07:18.185724 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.102.62:44777: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
I20260620 06:07:18.187335 19093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:07:18.187038 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.102.62:44777 (local address 127.11.102.3:51123)
0620 06:07:18.187212 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0620 06:07:18.187267 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.102.62:44777: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111}
I20260620 06:07:18.188829 19095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:07:18.188588 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.102.62:44777 (local address 127.11.102.2:52823)
0620 06:07:18.188712 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0620 06:07:18.188765 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.102.62:44777: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260620 06:07:18.188913 19094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:07:18.188415 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.102.62:44777 (local address 127.11.102.1:49795)
0620 06:07:18.188777 (+ 362us) negotiation.cc:107] Waiting for socket to connect
0620 06:07:18.188836 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.102.62:44777: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":266}
I20260620 06:07:18.189266 19128 mvcc.cc:205] Tried to move back new op lower bound from 7298808373980979200 to 7298808373826449408. Current Snapshot: MvccSnapshot[applied={T|T < 7298808373980979200 or (T in {7298808373980979200})}]
I20260620 06:07:18.192325 18367 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52524:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 06:07:18.192482 18367 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52524:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 06:07:18.196039 18367 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0a9e7156fd5c4d399486dc5932b407b2: Sending DeleteTablet for 3 replicas of tablet 1b84faf2ec3c4038bbfee0fac00de79f
I20260620 06:07:18.196647 18946 tablet_service.cc:1514] Processing DeleteTablet for tablet 1b84faf2ec3c4038bbfee0fac00de79f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 06:07:18 UTC) from {username='oryx', principal='oryx/127.11.102.61@KRBTEST.COM'} at 127.0.0.1:58698
I20260620 06:07:18.196740 18672 tablet_service.cc:1514] Processing DeleteTablet for tablet 1b84faf2ec3c4038bbfee0fac00de79f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 06:07:18 UTC) from {username='oryx', principal='oryx/127.11.102.61@KRBTEST.COM'} at 127.0.0.1:52238
I20260620 06:07:18.196801 18809 tablet_service.cc:1514] Processing DeleteTablet for tablet 1b84faf2ec3c4038bbfee0fac00de79f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 06:07:18 UTC) from {username='oryx', principal='oryx/127.11.102.61@KRBTEST.COM'} at 127.0.0.1:59314
I20260620 06:07:18.197062 19135 tablet_replica.cc:330] T 1b84faf2ec3c4038bbfee0fac00de79f P fb0783328ad3442fba39f260adf25308: stopping tablet replica
I20260620 06:07:18.197064 19134 tablet_replica.cc:330] T 1b84faf2ec3c4038bbfee0fac00de79f P d4b3fbe7b8664c968f75cc26b72410d7: stopping tablet replica
I20260620 06:07:18.197211 19134 raft_consensus.cc:2229] T 1b84faf2ec3c4038bbfee0fac00de79f P d4b3fbe7b8664c968f75cc26b72410d7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 06:07:18.197258 19135 raft_consensus.cc:2229] T 1b84faf2ec3c4038bbfee0fac00de79f P fb0783328ad3442fba39f260adf25308 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 06:07:18.197304 19136 tablet_replica.cc:330] T 1b84faf2ec3c4038bbfee0fac00de79f P f569ba0a0ea74d078dabe54eb42b9e21: stopping tablet replica
I20260620 06:07:18.197414 19134 raft_consensus.cc:2258] T 1b84faf2ec3c4038bbfee0fac00de79f P d4b3fbe7b8664c968f75cc26b72410d7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 06:07:18.197427 19136 raft_consensus.cc:2229] T 1b84faf2ec3c4038bbfee0fac00de79f P f569ba0a0ea74d078dabe54eb42b9e21 [term 1 LEADER]: Raft consensus shutting down.
I20260620 06:07:18.197468 19135 raft_consensus.cc:2258] T 1b84faf2ec3c4038bbfee0fac00de79f P fb0783328ad3442fba39f260adf25308 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 06:07:18.197633 19136 raft_consensus.cc:2258] T 1b84faf2ec3c4038bbfee0fac00de79f P f569ba0a0ea74d078dabe54eb42b9e21 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 06:07:18.197937 11672 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9HMp43/build/debug/bin/kudu with pid 18521
I20260620 06:07:18.199103 19136 ts_tablet_manager.cc:1905] T 1b84faf2ec3c4038bbfee0fac00de79f P f569ba0a0ea74d078dabe54eb42b9e21: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 06:07:18.202109 19136 ts_tablet_manager.cc:1918] T 1b84faf2ec3c4038bbfee0fac00de79f P f569ba0a0ea74d078dabe54eb42b9e21: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 06:07:18.202199 19136 log.cc:1196] T 1b84faf2ec3c4038bbfee0fac00de79f P f569ba0a0ea74d078dabe54eb42b9e21: Deleting WAL directory at /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935603120653-11672-0/minicluster-data/ts-1/wal/wals/1b84faf2ec3c4038bbfee0fac00de79f
I20260620 06:07:18.202497 19136 ts_tablet_manager.cc:1939] T 1b84faf2ec3c4038bbfee0fac00de79f P f569ba0a0ea74d078dabe54eb42b9e21: Deleting consensus metadata
I20260620 06:07:18.203253 18362 catalog_manager.cc:4841] TS f569ba0a0ea74d078dabe54eb42b9e21 (127.11.102.2:33643): tablet 1b84faf2ec3c4038bbfee0fac00de79f (table test-table [id=e6701a3312a642799248405742bdb48e]) successfully deleted
I20260620 06:07:18.204147 19134 ts_tablet_manager.cc:1905] T 1b84faf2ec3c4038bbfee0fac00de79f P d4b3fbe7b8664c968f75cc26b72410d7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 06:07:18.205605 19110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:07:18.205313 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.102.1:37681 (local address 127.0.0.1:52242)
0620 06:07:18.205453 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0620 06:07:18.205511 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.102.1:37681: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260620 06:07:18.205803 18362 catalog_manager.cc:4584] TS fb0783328ad3442fba39f260adf25308 (127.11.102.1:37681): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 1b84faf2ec3c4038bbfee0fac00de79f: Network error: Client connection negotiation failed: client connection to 127.11.102.1:37681: connect: Connection refused (error 111)
I20260620 06:07:18.205845 11672 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9HMp43/build/debug/bin/kudu with pid 18746
I20260620 06:07:18.208211 19134 ts_tablet_manager.cc:1918] T 1b84faf2ec3c4038bbfee0fac00de79f P d4b3fbe7b8664c968f75cc26b72410d7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 06:07:18.208279 19134 log.cc:1196] T 1b84faf2ec3c4038bbfee0fac00de79f P d4b3fbe7b8664c968f75cc26b72410d7: Deleting WAL directory at /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935603120653-11672-0/minicluster-data/ts-2/wal/wals/1b84faf2ec3c4038bbfee0fac00de79f
I20260620 06:07:18.208525 19134 ts_tablet_manager.cc:1939] T 1b84faf2ec3c4038bbfee0fac00de79f P d4b3fbe7b8664c968f75cc26b72410d7: Deleting consensus metadata
I20260620 06:07:18.209203 18361 catalog_manager.cc:4841] TS d4b3fbe7b8664c968f75cc26b72410d7 (127.11.102.3:36875): tablet 1b84faf2ec3c4038bbfee0fac00de79f (table test-table [id=e6701a3312a642799248405742bdb48e]) successfully deleted
I20260620 06:07:18.213011 11672 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9HMp43/build/debug/bin/kudu with pid 18882
I20260620 06:07:18.218576 11672 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9HMp43/build/debug/bin/kudu with pid 18238
I20260620 06:07:18.224278 11672 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9HMp43/build/debug/bin/kudu with pid 18346
2026-06-20T06:07:18Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (6960 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781935603120653-11672-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 06:07:18 dist-test-slave-9hm5 krb5kdc[19139](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 06:07:18 dist-test-slave-9hm5 krb5kdc[19139](info): set up 2 sockets
Jun 20 06:07:18 dist-test-slave-9hm5 krb5kdc[19139](info): commencing operation
krb5kdc: starting...
W20260620 06:07:19.256230 11672 mini_kdc.cc:121] Time spent starting KDC: real 1.009s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781935603120653-11672-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781935603120653-11672-0/krb5kdc/test-user.keytab.
Jun 20 06:07:19 dist-test-slave-9hm5 krb5kdc[19139](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935639, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T06:07:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T06:07:19Z Disabled control of system clock
WARNING: no policy specified for oryx/127.11.102.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.11.102.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.11.102.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781935603120653-11672-0/krb5kdc/oryx_127.11.102.62.keytab.
Entry for principal oryx/127.11.102.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781935603120653-11672-0/krb5kdc/oryx_127.11.102.62.keytab.
I20260620 06:07:19.374333 11672 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9HMp43/build/debug/bin/kudu
/tmp/dist-test-task9HMp43/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781935603120653-11672-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781935603120653-11672-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781935603120653-11672-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781935603120653-11672-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.102.62:36245
--webserver_interface=127.11.102.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781935603120653-11672-0/krb5kdc/oryx_127.11.102.62.keytab
--principal=oryx/127.11.102.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.102.20:35985
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.102.62:36245
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781935603120653-11672-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781935603120653-11672-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781935603120653-11672-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 06:07:19.481264 19152 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:07:19.481490 19152 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:07:19.481561 19152 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:07:19.485529 19152 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 06:07:19.485579 19152 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260620 06:07:19.485595 19152 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:07:19.485613 19152 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:07:19.485630 19152 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 06:07:19.485679 19152 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 06:07:19.490871 19152 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.102.20:35985
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781935603120653-11672-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781935603120653-11672-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.102.62:36245
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781935603120653-11672-0/krb5kdc/oryx_127.11.102.62.keytab
--principal=oryx/127.11.102.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.102.62:36245
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781935603120653-11672-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.102.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19152
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781935603120653-11672-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 05:28:48 UTC on ccycloud.quasar-fixhms.root.comops.site
build id 4457020
I20260620 06:07:19.491870 19152 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:07:19.492821 19152 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 06:07:19.500852 19157 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:07:19.500981 19152 server_base.cc:1034] running on GCE node
W20260620 06:07:19.500814 19160 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:07:19.500839 19158 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:07:19.501659 19152 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:07:19.502614 19152 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:07:19.503765 19152 hybrid_clock.cc:648] HybridClock initialized: now 1781935639503752 us; error 26 us; skew 500 ppm
Jun 20 06:07:19 dist-test-slave-9hm5 krb5kdc[19139](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935639, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.102.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 06:07:19.506342 19152 init.cc:377] Logged in from keytab as oryx/127.11.102.62@KRBTEST.COM (short username oryx)
I20260620 06:07:19.507192 19152 webserver.cc:460] Webserver started at http://127.11.102.62:34835/ using document root <none> and password file <none>
I20260620 06:07:19.507786 19152 fs_manager.cc:359] Metadata directory not provided
I20260620 06:07:19.507850 19152 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:07:19.508013 19152 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 06:07:19.509877 19152 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781935603120653-11672-0/minicluster-data/master-0/data/instance:
uuid: "05824e2e98474e8da81f776cc916c2e8"
format_stamp: "Formatted at 2026-06-20 06:07:19 on dist-test-slave-9hm5"
server_key: "d44c0ad674f6e730e7eed1ab5e9ef0eb"
server_key_iv: "d9112317a14ec372d5b91ed760d73dad"
server_key_version: "encryptionkey@0"
I20260620 06:07:19.510360 19152 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781935603120653-11672-0/minicluster-data/master-0/wal/instance:
uuid: "05824e2e98474e8da81f776cc916c2e8"
format_stamp: "Formatted at 2026-06-20 06:07:19 on dist-test-slave-9hm5"
server_key: "d44c0ad674f6e730e7eed1ab5e9ef0eb"
server_key_iv: "d9112317a14ec372d5b91ed760d73dad"
server_key_version: "encryptionkey@0"
I20260620 06:07:19.513775 19152 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260620 06:07:19.516017 19167 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:07:19.517216 19152 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 06:07:19.517375 19152 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781935603120653-11672-0/minicluster-data/master-0/data,/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781935603120653-11672-0/minicluster-data/master-0/wal
uuid: "05824e2e98474e8da81f776cc916c2e8"
format_stamp: "Formatted at 2026-06-20 06:07:19 on dist-test-slave-9hm5"
server_key: "d44c0ad674f6e730e7eed1ab5e9ef0eb"
server_key_iv: "d9112317a14ec372d5b91ed760d73dad"
server_key_version: "encryptionkey@0"
I20260620 06:07:19.517483 19152 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781935603120653-11672-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781935603120653-11672-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781935603120653-11672-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:07:19.542651 19152 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:07:19.545717 19152 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:07:19.545949 19152 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:07:19.553570 19152 rpc_server.cc:307] RPC server started. Bound to: 127.11.102.62:36245
I20260620 06:07:19.553586 19229 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.102.62:36245 every 8 connection(s)
I20260620 06:07:19.554489 19152 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781935603120653-11672-0/minicluster-data/master-0/data/info.pb
I20260620 06:07:19.557142 19230 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 06:07:19.559852 11672 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9HMp43/build/debug/bin/kudu as pid 19152
I20260620 06:07:19.560007 11672 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781935603120653-11672-0/minicluster-data/master-0/wal/instance
I20260620 06:07:19.560312 11672 external_mini_cluster.cc:1428] Setting key fe6620fc5edccd1acdc4fb8174b4dac1
I20260620 06:07:19.563009 19230 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 05824e2e98474e8da81f776cc916c2e8: Bootstrap starting.
I20260620 06:07:19.565582 19230 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 05824e2e98474e8da81f776cc916c2e8: Neither blocks nor log segments found. Creating new log.
I20260620 06:07:19.566361 19230 log.cc:824] T 00000000000000000000000000000000 P 05824e2e98474e8da81f776cc916c2e8: Log is configured to *not* fsync() on all Append() calls
I20260620 06:07:19.568614 19230 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 05824e2e98474e8da81f776cc916c2e8: No bootstrap required, opened a new log
Jun 20 06:07:19 dist-test-slave-9hm5 krb5kdc[19139](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935639, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.11.102.62@KRBTEST.COM
I20260620 06:07:19.573371 19230 raft_consensus.cc:348] T 00000000000000000000000000000000 P 05824e2e98474e8da81f776cc916c2e8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05824e2e98474e8da81f776cc916c2e8" member_type: VOTER last_known_addr { host: "127.11.102.62" port: 36245 } }
I20260620 06:07:19.573653 19230 raft_consensus.cc:374] T 00000000000000000000000000000000 P 05824e2e98474e8da81f776cc916c2e8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 06:07:19.573701 19230 raft_consensus.cc:729] T 00000000000000000000000000000000 P 05824e2e98474e8da81f776cc916c2e8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 05824e2e98474e8da81f776cc916c2e8, State: Initialized, Role: FOLLOWER
I20260620 06:07:19.574198 19230 consensus_queue.cc:260] T 00000000000000000000000000000000 P 05824e2e98474e8da81f776cc916c2e8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05824e2e98474e8da81f776cc916c2e8" member_type: VOTER last_known_addr { host: "127.11.102.62" port: 36245 } }
I20260620 06:07:19.574366 19230 raft_consensus.cc:388] T 00000000000000000000000000000000 P 05824e2e98474e8da81f776cc916c2e8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 06:07:19.574474 19230 raft_consensus.cc:482] T 00000000000000000000000000000000 P 05824e2e98474e8da81f776cc916c2e8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 06:07:19.574582 19230 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 05824e2e98474e8da81f776cc916c2e8 [term 0 FOLLOWER]: Advancing to term 1
I20260620 06:07:19.575701 19230 raft_consensus.cc:504] T 00000000000000000000000000000000 P 05824e2e98474e8da81f776cc916c2e8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05824e2e98474e8da81f776cc916c2e8" member_type: VOTER last_known_addr { host: "127.11.102.62" port: 36245 } }
I20260620 06:07:19.575771 19233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:07:19.561654 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56262 (local address 127.11.102.62:36245)
0620 06:07:19.562077 (+ 423us) server_negotiation.cc:206] Beginning negotiation
0620 06:07:19.562097 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0620 06:07:19.562139 (+ 42us) server_negotiation.cc:407] Connection header received
0620 06:07:19.562915 (+ 776us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:07:19.562936 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:07:19.563333 (+ 397us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:07:19.563702 (+ 369us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:07:19.564383 (+ 681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:07:19.565857 (+ 1474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:07:19.566859 (+ 1002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:07:19.567537 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:07:19.567730 (+ 193us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:07:19.570189 (+ 2459us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:07:19.570196 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:07:19.570214 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:07:19.570247 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:07:19.572454 (+ 2207us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:07:19.572887 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:07:19.572893 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:07:19.572898 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:07:19.573039 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:07:19.573294 (+ 255us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:07:19.573297 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:07:19.573299 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:07:19.573830 (+ 531us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:07:19.574023 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0620 06:07:19.574306 (+ 283us) server_negotiation.cc:299] Negotiation successful
0620 06:07:19.574768 (+ 462us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":269,"thread_start_us":118,"threads_started":1}
I20260620 06:07:19.576051 19230 leader_election.cc:302] T 00000000000000000000000000000000 P 05824e2e98474e8da81f776cc916c2e8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 05824e2e98474e8da81f776cc916c2e8; no voters:
I20260620 06:07:19.576381 19230 leader_election.cc:288] T 00000000000000000000000000000000 P 05824e2e98474e8da81f776cc916c2e8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 06:07:19.576505 19235 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 05824e2e98474e8da81f776cc916c2e8 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 06:07:19.576705 19235 raft_consensus.cc:686] T 00000000000000000000000000000000 P 05824e2e98474e8da81f776cc916c2e8 [term 1 LEADER]: Becoming Leader. State: Replica: 05824e2e98474e8da81f776cc916c2e8, State: Running, Role: LEADER
I20260620 06:07:19.577059 19235 consensus_queue.cc:237] T 00000000000000000000000000000000 P 05824e2e98474e8da81f776cc916c2e8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05824e2e98474e8da81f776cc916c2e8" member_type: VOTER last_known_addr { host: "127.11.102.62" port: 36245 } }
I20260620 06:07:19.577652 19230 sys_catalog.cc:564] T 00000000000000000000000000000000 P 05824e2e98474e8da81f776cc916c2e8 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 06:07:19.578725 19236 sys_catalog.cc:455] T 00000000000000000000000000000000 P 05824e2e98474e8da81f776cc916c2e8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "05824e2e98474e8da81f776cc916c2e8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05824e2e98474e8da81f776cc916c2e8" member_type: VOTER last_known_addr { host: "127.11.102.62" port: 36245 } } }
I20260620 06:07:19.578706 19237 sys_catalog.cc:455] T 00000000000000000000000000000000 P 05824e2e98474e8da81f776cc916c2e8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 05824e2e98474e8da81f776cc916c2e8. Latest consensus state: current_term: 1 leader_uuid: "05824e2e98474e8da81f776cc916c2e8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05824e2e98474e8da81f776cc916c2e8" member_type: VOTER last_known_addr { host: "127.11.102.62" port: 36245 } } }
I20260620 06:07:19.578850 19236 sys_catalog.cc:458] T 00000000000000000000000000000000 P 05824e2e98474e8da81f776cc916c2e8 [sys.catalog]: This master's current role is: LEADER
I20260620 06:07:19.578850 19237 sys_catalog.cc:458] T 00000000000000000000000000000000 P 05824e2e98474e8da81f776cc916c2e8 [sys.catalog]: This master's current role is: LEADER
I20260620 06:07:19.579219 19244 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 06:07:19.582296 19244 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 06:07:19.587889 19244 catalog_manager.cc:1347] Generated new cluster ID: f57c5e1f4f2d4170b3b08bb5af6acf85
I20260620 06:07:19.588024 19244 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 06:07:19.602890 19244 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 06:07:19.603885 19244 catalog_manager.cc:1504] Loading token signing keys...
I20260620 06:07:19.608527 19244 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 05824e2e98474e8da81f776cc916c2e8: Generated new TSK 0
I20260620 06:07:19.609174 19244 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.11.102.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.11.102.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.11.102.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781935603120653-11672-0/krb5kdc/oryx_127.11.102.1.keytab.
Entry for principal oryx/127.11.102.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781935603120653-11672-0/krb5kdc/oryx_127.11.102.1.keytab.
I20260620 06:07:19.645391 11672 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9HMp43/build/debug/bin/kudu
/tmp/dist-test-task9HMp43/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781935603120653-11672-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781935603120653-11672-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781935603120653-11672-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781935603120653-11672-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.102.1:0
--local_ip_for_outbound_sockets=127.11.102.1
--webserver_interface=127.11.102.1
--webserver_port=0
--tserver_master_addrs=127.11.102.62:36245
--keytab_file=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781935603120653-11672-0/krb5kdc/oryx_127.11.102.1.keytab
--principal=oryx/127.11.102.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.102.20:35985
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781935603120653-11672-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781935603120653-11672-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781935603120653-11672-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 06:07:19.769150 19256 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:07:19.769387 19256 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:07:19.769428 19256 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:07:19.773486 19256 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:07:19.773540 19256 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:07:19.773577 19256 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 06:07:19.773629 19256 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.102.1
I20260620 06:07:19.778816 19256 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.102.20:35985
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781935603120653-11672-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781935603120653-11672-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781935603120653-11672-0/krb5kdc/oryx_127.11.102.1.keytab
--principal=oryx/127.11.102.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.102.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781935603120653-11672-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.11.102.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.11.102.62:36245
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19256
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.102.1
--log_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781935603120653-11672-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 05:28:48 UTC on ccycloud.quasar-fixhms.root.comops.site
build id 4457020
I20260620 06:07:19.779909 19256 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:07:19.780902 19256 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 06:07:19.790071 19261 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:07:19.790211 19256 server_base.cc:1034] running on GCE node
W20260620 06:07:19.790118 19262 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:07:19.790128 19264 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:07:19.790706 19256 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:07:19.791324 19256 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:07:19.792507 19256 hybrid_clock.cc:648] HybridClock initialized: now 1781935639792473 us; error 57 us; skew 500 ppm
Jun 20 06:07:19 dist-test-slave-9hm5 krb5kdc[19139](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935639, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.102.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 06:07:19.795213 19256 init.cc:377] Logged in from keytab as oryx/127.11.102.1@KRBTEST.COM (short username oryx)
I20260620 06:07:19.796098 19256 webserver.cc:460] Webserver started at http://127.11.102.1:44763/ using document root <none> and password file <none>
I20260620 06:07:19.796639 19256 fs_manager.cc:359] Metadata directory not provided
I20260620 06:07:19.796726 19256 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:07:19.796965 19256 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 06:07:19.799130 19256 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781935603120653-11672-0/minicluster-data/ts-0/data/instance:
uuid: "7fe86b816edb423498c5dd937c81279c"
format_stamp: "Formatted at 2026-06-20 06:07:19 on dist-test-slave-9hm5"
server_key: "0d97b4d8c61b85a74b48c33fdf95f580"
server_key_iv: "41a66c2bf759b7e056c264345020c989"
server_key_version: "encryptionkey@0"
I20260620 06:07:19.799759 19256 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781935603120653-11672-0/minicluster-data/ts-0/wal/instance:
uuid: "7fe86b816edb423498c5dd937c81279c"
format_stamp: "Formatted at 2026-06-20 06:07:19 on dist-test-slave-9hm5"
server_key: "0d97b4d8c61b85a74b48c33fdf95f580"
server_key_iv: "41a66c2bf759b7e056c264345020c989"
server_key_version: "encryptionkey@0"
I20260620 06:07:19.803232 19256 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260620 06:07:19.805641 19271 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:07:19.806825 19256 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 06:07:19.806951 19256 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781935603120653-11672-0/minicluster-data/ts-0/data,/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781935603120653-11672-0/minicluster-data/ts-0/wal
uuid: "7fe86b816edb423498c5dd937c81279c"
format_stamp: "Formatted at 2026-06-20 06:07:19 on dist-test-slave-9hm5"
server_key: "0d97b4d8c61b85a74b48c33fdf95f580"
server_key_iv: "41a66c2bf759b7e056c264345020c989"
server_key_version: "encryptionkey@0"
I20260620 06:07:19.807051 19256 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781935603120653-11672-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781935603120653-11672-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781935603120653-11672-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:07:19.817168 19256 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:07:19.819911 19256 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:07:19.820088 19256 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:07:19.821403 19256 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 06:07:19.821470 19256 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 06:07:19.821542 19256 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 06:07:19.821599 19256 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 06:07:19 dist-test-slave-9hm5 krb5kdc[19139](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935639, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.102.1@KRBTEST.COM for oryx/127.11.102.62@KRBTEST.COM
I20260620 06:07:19.832336 19256 rpc_server.cc:307] RPC server started. Bound to: 127.11.102.1:33553
I20260620 06:07:19.832412 19386 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.102.1:33553 every 8 connection(s)
I20260620 06:07:19.833190 19256 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781935603120653-11672-0/minicluster-data/ts-0/data/info.pb
I20260620 06:07:19.834039 11672 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9HMp43/build/debug/bin/kudu as pid 19256
I20260620 06:07:19.834182 11672 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781935603120653-11672-0/minicluster-data/ts-0/wal/instance
I20260620 06:07:19.834563 11672 external_mini_cluster.cc:1428] Setting key 27bd9ef2ec31af8d6162e915f5bfdfaa
I20260620 06:07:19.837142 19280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:07:19.822839 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.102.62:36245 (local address 127.11.102.1:33737)
0620 06:07:19.823294 (+ 455us) negotiation.cc:107] Waiting for socket to connect
0620 06:07:19.823333 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0620 06:07:19.824389 (+ 1056us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:07:19.824977 (+ 588us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:07:19.824985 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:07:19.825331 (+ 346us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:07:19.825962 (+ 631us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:07:19.825984 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:07:19.827134 (+ 1150us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:07:19.827140 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:07:19.828715 (+ 1575us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:07:19.828728 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:07:19.829402 (+ 674us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:07:19.829406 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:07:19.829589 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:07:19.830165 (+ 576us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:07:19.830184 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:07:19.832017 (+ 1833us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:07:19.834160 (+ 2143us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:07:19.834165 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:07:19.834178 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:07:19.834533 (+ 355us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:07:19.834839 (+ 306us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:07:19.834842 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:07:19.834844 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:07:19.835007 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:07:19.835458 (+ 451us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:07:19.835463 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:07:19.835772 (+ 309us) client_negotiation.cc:769] Sending connection context
0620 06:07:19.835942 (+ 170us) client_negotiation.cc:240] Negotiation successful
0620 06:07:19.836247 (+ 305us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":280,"thread_start_us":129,"threads_started":1}
I20260620 06:07:19.837983 19233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:07:19.822320 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.102.1:33737 (local address 127.11.102.62:36245)
0620 06:07:19.822442 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0620 06:07:19.822445 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 06:07:19.823432 (+ 987us) server_negotiation.cc:407] Connection header received
0620 06:07:19.824634 (+ 1202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:07:19.824638 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:07:19.824698 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:07:19.824802 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:07:19.826101 (+ 1299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:07:19.826984 (+ 883us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:07:19.828846 (+ 1862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:07:19.829221 (+ 375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:07:19.829358 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:07:19.832237 (+ 2879us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:07:19.832246 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:07:19.832248 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:07:19.832272 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:07:19.834032 (+ 1760us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:07:19.834623 (+ 591us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:07:19.834628 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:07:19.834633 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:07:19.834729 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:07:19.835120 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:07:19.835124 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:07:19.835126 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:07:19.835353 (+ 227us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:07:19.835464 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0620 06:07:19.837727 (+ 2263us) server_negotiation.cc:299] Negotiation successful
0620 06:07:19.837843 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260620 06:07:19.842998 19387 heartbeater.cc:347] Connected to a master server at 127.11.102.62:36245
I20260620 06:07:19.843154 19387 heartbeater.cc:464] Registering TS with master...
I20260620 06:07:19.843562 19387 heartbeater.cc:511] Master 127.11.102.62:36245 requested a full tablet report, sending...
I20260620 06:07:19.845068 19183 ts_manager.cc:194] Registered new tserver with Master: 7fe86b816edb423498c5dd937c81279c (127.11.102.1:33553)
I20260620 06:07:19.846484 19183 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.11.102.1@KRBTEST.COM'} at 127.11.102.1:33737
I20260620 06:07:19.850764 19394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:07:19.841396 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.102.62:36245 (local address 127.11.102.1:39893)
0620 06:07:19.841790 (+ 394us) negotiation.cc:107] Waiting for socket to connect
0620 06:07:19.841804 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 06:07:19.841922 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:07:19.842212 (+ 290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:07:19.842215 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:07:19.842414 (+ 199us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:07:19.842539 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:07:19.842547 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:07:19.843588 (+ 1041us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:07:19.843592 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:07:19.844790 (+ 1198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:07:19.844803 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:07:19.845587 (+ 784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:07:19.845593 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:07:19.845720 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:07:19.846939 (+ 1219us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:07:19.846951 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:07:19.847410 (+ 459us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:07:19.849428 (+ 2018us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:07:19.849431 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:07:19.849432 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:07:19.849617 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:07:19.849909 (+ 292us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:07:19.849911 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:07:19.849913 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:07:19.849982 (+ 69us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:07:19.850410 (+ 428us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:07:19.850411 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:07:19.850475 (+ 64us) client_negotiation.cc:769] Sending connection context
0620 06:07:19.850550 (+ 75us) client_negotiation.cc:240] Negotiation successful
0620 06:07:19.850639 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":326,"thread_start_us":99,"threads_started":1}
WARNING: no policy specified for oryx/127.11.102.2@KRBTEST.COM; defaulting to no policy
I20260620 06:07:19.850934 19233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:07:19.841210 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.102.1:39893 (local address 127.11.102.62:36245)
0620 06:07:19.841318 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0620 06:07:19.841321 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 06:07:19.841864 (+ 543us) server_negotiation.cc:407] Connection header received
0620 06:07:19.842011 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:07:19.842014 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:07:19.842053 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:07:19.842097 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:07:19.842660 (+ 563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:07:19.843424 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:07:19.844925 (+ 1501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:07:19.845335 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:07:19.845423 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:07:19.847534 (+ 2111us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:07:19.847540 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:07:19.847545 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:07:19.847575 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:07:19.849317 (+ 1742us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:07:19.849721 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:07:19.849725 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:07:19.849727 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:07:19.849813 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:07:19.850069 (+ 256us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:07:19.850076 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:07:19.850080 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:07:19.850312 (+ 232us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:07:19.850418 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0620 06:07:19.850624 (+ 206us) server_negotiation.cc:299] Negotiation successful
0620 06:07:19.850737 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.11.102.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.11.102.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781935603120653-11672-0/krb5kdc/oryx_127.11.102.2.keytab.
Entry for principal oryx/127.11.102.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781935603120653-11672-0/krb5kdc/oryx_127.11.102.2.keytab.
I20260620 06:07:19.866761 11672 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9HMp43/build/debug/bin/kudu
/tmp/dist-test-task9HMp43/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781935603120653-11672-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781935603120653-11672-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781935603120653-11672-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781935603120653-11672-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.102.2:0
--local_ip_for_outbound_sockets=127.11.102.2
--webserver_interface=127.11.102.2
--webserver_port=0
--tserver_master_addrs=127.11.102.62:36245
--keytab_file=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781935603120653-11672-0/krb5kdc/oryx_127.11.102.2.keytab
--principal=oryx/127.11.102.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.102.20:35985
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781935603120653-11672-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781935603120653-11672-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781935603120653-11672-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 06:07:19.974975 19396 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:07:19.975229 19396 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:07:19.975270 19396 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:07:19.979523 19396 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:07:19.979578 19396 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:07:19.979671 19396 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 06:07:19.979761 19396 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.102.2
I20260620 06:07:19.984879 19396 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.102.20:35985
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781935603120653-11672-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781935603120653-11672-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781935603120653-11672-0/krb5kdc/oryx_127.11.102.2.keytab
--principal=oryx/127.11.102.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.102.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781935603120653-11672-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.11.102.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.11.102.62:36245
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19396
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.102.2
--log_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781935603120653-11672-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 05:28:48 UTC on ccycloud.quasar-fixhms.root.comops.site
build id 4457020
I20260620 06:07:19.985867 19396 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:07:19.986877 19396 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 06:07:19.996284 19404 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:07:19.996304 19402 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:07:19.996304 19401 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:07:19.996354 19396 server_base.cc:1034] running on GCE node
I20260620 06:07:19.996912 19396 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:07:19.997509 19396 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:07:19.998692 19396 hybrid_clock.cc:648] HybridClock initialized: now 1781935639998680 us; error 26 us; skew 500 ppm
Jun 20 06:07:19 dist-test-slave-9hm5 krb5kdc[19139](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935639, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.102.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 06:07:20.001420 19396 init.cc:377] Logged in from keytab as oryx/127.11.102.2@KRBTEST.COM (short username oryx)
I20260620 06:07:20.002350 19396 webserver.cc:460] Webserver started at http://127.11.102.2:34875/ using document root <none> and password file <none>
I20260620 06:07:20.002892 19396 fs_manager.cc:359] Metadata directory not provided
I20260620 06:07:20.002949 19396 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:07:20.003181 19396 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 06:07:20.005167 19396 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781935603120653-11672-0/minicluster-data/ts-1/data/instance:
uuid: "4fe4e45f6e02471594ff656d8aff63ac"
format_stamp: "Formatted at 2026-06-20 06:07:19 on dist-test-slave-9hm5"
server_key: "0a6cb7bed14c1be696d052d0d1da7e1f"
server_key_iv: "4ecea4d162c8701739d89d41833cf1f9"
server_key_version: "encryptionkey@0"
I20260620 06:07:20.005712 19396 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781935603120653-11672-0/minicluster-data/ts-1/wal/instance:
uuid: "4fe4e45f6e02471594ff656d8aff63ac"
format_stamp: "Formatted at 2026-06-20 06:07:19 on dist-test-slave-9hm5"
server_key: "0a6cb7bed14c1be696d052d0d1da7e1f"
server_key_iv: "4ecea4d162c8701739d89d41833cf1f9"
server_key_version: "encryptionkey@0"
I20260620 06:07:20.009229 19396 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.005s
I20260620 06:07:20.011660 19411 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:07:20.012861 19396 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.002s
I20260620 06:07:20.013001 19396 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781935603120653-11672-0/minicluster-data/ts-1/data,/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781935603120653-11672-0/minicluster-data/ts-1/wal
uuid: "4fe4e45f6e02471594ff656d8aff63ac"
format_stamp: "Formatted at 2026-06-20 06:07:19 on dist-test-slave-9hm5"
server_key: "0a6cb7bed14c1be696d052d0d1da7e1f"
server_key_iv: "4ecea4d162c8701739d89d41833cf1f9"
server_key_version: "encryptionkey@0"
I20260620 06:07:20.013106 19396 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781935603120653-11672-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781935603120653-11672-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781935603120653-11672-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:07:20.037055 19396 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:07:20.040277 19396 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:07:20.040470 19396 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:07:20.041934 19396 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 06:07:20.042004 19396 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 06:07:20.042075 19396 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 06:07:20.042129 19396 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 06:07:20 dist-test-slave-9hm5 krb5kdc[19139](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935639, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.102.2@KRBTEST.COM for oryx/127.11.102.62@KRBTEST.COM
I20260620 06:07:20.052989 19396 rpc_server.cc:307] RPC server started. Bound to: 127.11.102.2:38643
I20260620 06:07:20.053298 19526 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.102.2:38643 every 8 connection(s)
I20260620 06:07:20.054097 19396 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781935603120653-11672-0/minicluster-data/ts-1/data/info.pb
I20260620 06:07:20.056313 19233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:07:20.042724 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.102.2:48821 (local address 127.11.102.62:36245)
0620 06:07:20.042855 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0620 06:07:20.042859 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 06:07:20.043875 (+ 1016us) server_negotiation.cc:407] Connection header received
0620 06:07:20.044944 (+ 1069us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:07:20.044948 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:07:20.045013 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:07:20.045093 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:07:20.046596 (+ 1503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:07:20.047501 (+ 905us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:07:20.049273 (+ 1772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:07:20.049667 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:07:20.049793 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:07:20.052600 (+ 2807us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:07:20.052606 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:07:20.052612 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:07:20.052647 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:07:20.054306 (+ 1659us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:07:20.054879 (+ 573us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:07:20.054882 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:07:20.054884 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:07:20.054967 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:07:20.055259 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:07:20.055262 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:07:20.055263 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:07:20.055422 (+ 159us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:07:20.055496 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0620 06:07:20.056095 (+ 599us) server_negotiation.cc:299] Negotiation successful
0620 06:07:20.056198 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260620 06:07:20.056965 19420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:07:20.043201 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.102.62:36245 (local address 127.11.102.2:48821)
0620 06:07:20.043735 (+ 534us) negotiation.cc:107] Waiting for socket to connect
0620 06:07:20.043776 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0620 06:07:20.044689 (+ 913us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:07:20.045555 (+ 866us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:07:20.045569 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:07:20.045936 (+ 367us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:07:20.046443 (+ 507us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:07:20.046460 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:07:20.047670 (+ 1210us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:07:20.047673 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:07:20.049125 (+ 1452us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:07:20.049138 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:07:20.049892 (+ 754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:07:20.049898 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:07:20.050110 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:07:20.050678 (+ 568us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:07:20.050694 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:07:20.052437 (+ 1743us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:07:20.054410 (+ 1973us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:07:20.054414 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:07:20.054424 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:07:20.054757 (+ 333us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:07:20.055045 (+ 288us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:07:20.055048 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:07:20.055049 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:07:20.055174 (+ 125us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:07:20.055509 (+ 335us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:07:20.055513 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:07:20.055804 (+ 291us) client_negotiation.cc:769] Sending connection context
0620 06:07:20.055974 (+ 170us) client_negotiation.cc:240] Negotiation successful
0620 06:07:20.056199 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":281,"thread_start_us":131,"threads_started":1}
I20260620 06:07:20.058073 19527 heartbeater.cc:347] Connected to a master server at 127.11.102.62:36245
I20260620 06:07:20.058228 19527 heartbeater.cc:464] Registering TS with master...
I20260620 06:07:20.058660 19527 heartbeater.cc:511] Master 127.11.102.62:36245 requested a full tablet report, sending...
I20260620 06:07:20.059572 19184 ts_manager.cc:194] Registered new tserver with Master: 4fe4e45f6e02471594ff656d8aff63ac (127.11.102.2:38643)
I20260620 06:07:20.060195 19184 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.11.102.2@KRBTEST.COM'} at 127.11.102.2:48821
I20260620 06:07:20.062674 11672 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9HMp43/build/debug/bin/kudu as pid 19396
I20260620 06:07:20.062819 11672 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781935603120653-11672-0/minicluster-data/ts-1/wal/instance
I20260620 06:07:20.063125 11672 external_mini_cluster.cc:1428] Setting key 20469d94fb6631ccbcfa78fafbf05435
I20260620 06:07:20.067389 19233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:07:20.060293 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.102.2:38123 (local address 127.11.102.62:36245)
0620 06:07:20.060398 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0620 06:07:20.060403 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 06:07:20.060584 (+ 181us) server_negotiation.cc:407] Connection header received
0620 06:07:20.060655 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:07:20.060658 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:07:20.060702 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:07:20.060782 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:07:20.061238 (+ 456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:07:20.061692 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:07:20.062579 (+ 887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:07:20.062894 (+ 315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:07:20.062984 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:07:20.064080 (+ 1096us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:07:20.064083 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:07:20.064086 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:07:20.064110 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:07:20.065502 (+ 1392us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:07:20.065963 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:07:20.065969 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:07:20.065970 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:07:20.066055 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:07:20.066324 (+ 269us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:07:20.066329 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:07:20.066331 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:07:20.066633 (+ 302us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:07:20.066745 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0620 06:07:20.067109 (+ 364us) server_negotiation.cc:299] Negotiation successful
0620 06:07:20.067259 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260620 06:07:20.067473 19533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:07:20.060243 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.102.62:36245 (local address 127.11.102.2:38123)
0620 06:07:20.060503 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0620 06:07:20.060515 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 06:07:20.060590 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:07:20.060813 (+ 223us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:07:20.060816 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:07:20.061010 (+ 194us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:07:20.061133 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:07:20.061141 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:07:20.061814 (+ 673us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:07:20.061817 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:07:20.062455 (+ 638us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:07:20.062463 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:07:20.062988 (+ 525us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:07:20.062991 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:07:20.063091 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:07:20.063589 (+ 498us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:07:20.063600 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:07:20.063948 (+ 348us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:07:20.065629 (+ 1681us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:07:20.065635 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:07:20.065639 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:07:20.065860 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:07:20.066153 (+ 293us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:07:20.066156 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:07:20.066158 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:07:20.066236 (+ 78us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:07:20.066750 (+ 514us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:07:20.066754 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:07:20.066866 (+ 112us) client_negotiation.cc:769] Sending connection context
0620 06:07:20.066976 (+ 110us) client_negotiation.cc:240] Negotiation successful
0620 06:07:20.067177 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":177,"mutex_wait_us":1,"thread_start_us":111,"threads_started":1}
WARNING: no policy specified for oryx/127.11.102.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.11.102.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.11.102.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781935603120653-11672-0/krb5kdc/oryx_127.11.102.3.keytab.
Entry for principal oryx/127.11.102.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781935603120653-11672-0/krb5kdc/oryx_127.11.102.3.keytab.
I20260620 06:07:20.088927 11672 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9HMp43/build/debug/bin/kudu
/tmp/dist-test-task9HMp43/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781935603120653-11672-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781935603120653-11672-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781935603120653-11672-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781935603120653-11672-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.102.3:0
--local_ip_for_outbound_sockets=127.11.102.3
--webserver_interface=127.11.102.3
--webserver_port=0
--tserver_master_addrs=127.11.102.62:36245
--keytab_file=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781935603120653-11672-0/krb5kdc/oryx_127.11.102.3.keytab
--principal=oryx/127.11.102.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.102.20:35985
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781935603120653-11672-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781935603120653-11672-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781935603120653-11672-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 06:07:20.196852 19536 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:07:20.197121 19536 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:07:20.197193 19536 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:07:20.201187 19536 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:07:20.201267 19536 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:07:20.201332 19536 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 06:07:20.201416 19536 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.102.3
I20260620 06:07:20.206537 19536 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.102.20:35985
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781935603120653-11672-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781935603120653-11672-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781935603120653-11672-0/krb5kdc/oryx_127.11.102.3.keytab
--principal=oryx/127.11.102.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.102.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781935603120653-11672-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.11.102.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.11.102.62:36245
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19536
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.102.3
--log_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781935603120653-11672-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 05:28:48 UTC on ccycloud.quasar-fixhms.root.comops.site
build id 4457020
I20260620 06:07:20.207585 19536 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:07:20.208652 19536 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 06:07:20.218290 19544 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:07:20.218287 19541 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:07:20.218302 19542 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:07:20.218515 19536 server_base.cc:1034] running on GCE node
I20260620 06:07:20.219013 19536 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:07:20.219563 19536 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:07:20.220757 19536 hybrid_clock.cc:648] HybridClock initialized: now 1781935640220746 us; error 27 us; skew 500 ppm
Jun 20 06:07:20 dist-test-slave-9hm5 krb5kdc[19139](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935640, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.102.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 06:07:20.223694 19536 init.cc:377] Logged in from keytab as oryx/127.11.102.3@KRBTEST.COM (short username oryx)
I20260620 06:07:20.224481 19536 webserver.cc:460] Webserver started at http://127.11.102.3:39943/ using document root <none> and password file <none>
I20260620 06:07:20.225018 19536 fs_manager.cc:359] Metadata directory not provided
I20260620 06:07:20.225106 19536 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:07:20.225328 19536 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 06:07:20.227281 19536 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781935603120653-11672-0/minicluster-data/ts-2/data/instance:
uuid: "2316ce59fd744c01a82cb798c635b913"
format_stamp: "Formatted at 2026-06-20 06:07:20 on dist-test-slave-9hm5"
server_key: "835418d058f98627301c9ac62e1b00fe"
server_key_iv: "9faa95c60b1558278fe60e06c579c994"
server_key_version: "encryptionkey@0"
I20260620 06:07:20.227851 19536 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781935603120653-11672-0/minicluster-data/ts-2/wal/instance:
uuid: "2316ce59fd744c01a82cb798c635b913"
format_stamp: "Formatted at 2026-06-20 06:07:20 on dist-test-slave-9hm5"
server_key: "835418d058f98627301c9ac62e1b00fe"
server_key_iv: "9faa95c60b1558278fe60e06c579c994"
server_key_version: "encryptionkey@0"
I20260620 06:07:20.231298 19536 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260620 06:07:20.233644 19551 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:07:20.234824 19536 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 06:07:20.234957 19536 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781935603120653-11672-0/minicluster-data/ts-2/data,/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781935603120653-11672-0/minicluster-data/ts-2/wal
uuid: "2316ce59fd744c01a82cb798c635b913"
format_stamp: "Formatted at 2026-06-20 06:07:20 on dist-test-slave-9hm5"
server_key: "835418d058f98627301c9ac62e1b00fe"
server_key_iv: "9faa95c60b1558278fe60e06c579c994"
server_key_version: "encryptionkey@0"
I20260620 06:07:20.235066 19536 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781935603120653-11672-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781935603120653-11672-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781935603120653-11672-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:07:20.252194 19536 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:07:20.255342 19536 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:07:20.255540 19536 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:07:20.256852 19536 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 06:07:20.256927 19536 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 06:07:20.256994 19536 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 06:07:20.257030 19536 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 06:07:20 dist-test-slave-9hm5 krb5kdc[19139](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935640, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.102.3@KRBTEST.COM for oryx/127.11.102.62@KRBTEST.COM
I20260620 06:07:20.267939 19536 rpc_server.cc:307] RPC server started. Bound to: 127.11.102.3:37845
I20260620 06:07:20.268057 19666 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.102.3:37845 every 8 connection(s)
I20260620 06:07:20.268993 19536 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781935603120653-11672-0/minicluster-data/ts-2/data/info.pb
I20260620 06:07:20.270624 19233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:07:20.257808 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.102.3:44177 (local address 127.11.102.62:36245)
0620 06:07:20.257926 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0620 06:07:20.257929 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 06:07:20.259045 (+ 1116us) server_negotiation.cc:407] Connection header received
0620 06:07:20.260299 (+ 1254us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:07:20.260303 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:07:20.260385 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:07:20.260494 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:07:20.261841 (+ 1347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:07:20.262516 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:07:20.263820 (+ 1304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:07:20.264125 (+ 305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:07:20.264254 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:07:20.266918 (+ 2664us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:07:20.266922 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:07:20.266925 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:07:20.266958 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:07:20.268639 (+ 1681us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:07:20.269123 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:07:20.269127 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:07:20.269129 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:07:20.269217 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:07:20.269677 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:07:20.269681 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:07:20.269683 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:07:20.269896 (+ 213us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:07:20.270001 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0620 06:07:20.270390 (+ 389us) server_negotiation.cc:299] Negotiation successful
0620 06:07:20.270513 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260620 06:07:20.271342 19560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:07:20.258303 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.102.62:36245 (local address 127.11.102.3:44177)
0620 06:07:20.258892 (+ 589us) negotiation.cc:107] Waiting for socket to connect
0620 06:07:20.258931 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0620 06:07:20.260027 (+ 1096us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:07:20.260765 (+ 738us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:07:20.260775 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:07:20.261092 (+ 317us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:07:20.261684 (+ 592us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:07:20.261701 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:07:20.262639 (+ 938us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:07:20.262642 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:07:20.263687 (+ 1045us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:07:20.263697 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:07:20.264297 (+ 600us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:07:20.264301 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:07:20.264490 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:07:20.265035 (+ 545us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:07:20.265053 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:07:20.266460 (+ 1407us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:07:20.268745 (+ 2285us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:07:20.268749 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:07:20.268759 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:07:20.268996 (+ 237us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:07:20.269372 (+ 376us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:07:20.269378 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:07:20.269380 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:07:20.269553 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:07:20.269997 (+ 444us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:07:20.270001 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:07:20.270195 (+ 194us) client_negotiation.cc:769] Sending connection context
0620 06:07:20.270342 (+ 147us) client_negotiation.cc:240] Negotiation successful
0620 06:07:20.270579 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":308,"thread_start_us":137,"threads_started":1}
I20260620 06:07:20.272497 19667 heartbeater.cc:347] Connected to a master server at 127.11.102.62:36245
I20260620 06:07:20.272641 19667 heartbeater.cc:464] Registering TS with master...
I20260620 06:07:20.273087 19667 heartbeater.cc:511] Master 127.11.102.62:36245 requested a full tablet report, sending...
I20260620 06:07:20.273986 19183 ts_manager.cc:194] Registered new tserver with Master: 2316ce59fd744c01a82cb798c635b913 (127.11.102.3:37845)
I20260620 06:07:20.274691 11672 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9HMp43/build/debug/bin/kudu as pid 19536
I20260620 06:07:20.274700 19183 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.11.102.3@KRBTEST.COM'} at 127.11.102.3:44177
I20260620 06:07:20.274814 11672 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781935603120653-11672-0/minicluster-data/ts-2/wal/instance
I20260620 06:07:20.275154 11672 external_mini_cluster.cc:1428] Setting key a97e32fa72d3ac0d1a36b0ec04312ad4
I20260620 06:07:20.277130 11672 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 06:07:20.281796 19673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:07:20.274425 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.102.62:36245 (local address 127.11.102.3:55069)
0620 06:07:20.274845 (+ 420us) negotiation.cc:107] Waiting for socket to connect
0620 06:07:20.274861 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 06:07:20.274983 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:07:20.275250 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:07:20.275252 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:07:20.275454 (+ 202us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:07:20.275639 (+ 185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:07:20.275647 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:07:20.276561 (+ 914us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:07:20.276564 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:07:20.277242 (+ 678us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:07:20.277254 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:07:20.277697 (+ 443us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:07:20.277700 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:07:20.277783 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:07:20.278251 (+ 468us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:07:20.278264 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:07:20.278599 (+ 335us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:07:20.280392 (+ 1793us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:07:20.280394 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:07:20.280396 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:07:20.280594 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:07:20.280869 (+ 275us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:07:20.280871 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:07:20.280873 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:07:20.280933 (+ 60us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:07:20.281357 (+ 424us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:07:20.281359 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:07:20.281421 (+ 62us) client_negotiation.cc:769] Sending connection context
0620 06:07:20.281495 (+ 74us) client_negotiation.cc:240] Negotiation successful
0620 06:07:20.281586 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":345,"mutex_wait_us":3,"thread_start_us":111,"threads_started":1}
I20260620 06:07:20.281912 19233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:07:20.274577 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.102.3:55069 (local address 127.11.102.62:36245)
0620 06:07:20.274682 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0620 06:07:20.274686 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 06:07:20.274922 (+ 236us) server_negotiation.cc:407] Connection header received
0620 06:07:20.275086 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:07:20.275091 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:07:20.275134 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:07:20.275218 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:07:20.275744 (+ 526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:07:20.276446 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:07:20.277367 (+ 921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:07:20.277591 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:07:20.277696 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:07:20.278751 (+ 1055us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:07:20.278755 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:07:20.278757 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:07:20.278780 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:07:20.280286 (+ 1506us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:07:20.280688 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:07:20.280694 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:07:20.280699 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:07:20.280786 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:07:20.281020 (+ 234us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:07:20.281025 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:07:20.281030 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:07:20.281255 (+ 225us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:07:20.281367 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0620 06:07:20.281582 (+ 215us) server_negotiation.cc:299] Negotiation successful
0620 06:07:20.281697 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
Jun 20 06:07:20 dist-test-slave-9hm5 krb5kdc[19139](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935640, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 20 06:07:20 dist-test-slave-9hm5 krb5kdc[19139](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.11.102.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 06:07:20 dist-test-slave-9hm5 krb5kdc[19139](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.11.102.62@KRBTEST.COM, Server not found in Kerberos database
W20260620 06:07:20.300913 19233 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 06:07:20.295056 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56274 (local address 127.11.102.62:36245)
0620 06:07:20.295172 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0620 06:07:20.295175 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 06:07:20.295186 (+ 11us) server_negotiation.cc:407] Connection header received
0620 06:07:20.295214 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:07:20.295215 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:07:20.295246 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:07:20.295351 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:07:20.296077 (+ 726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:07:20.296567 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:07:20.297394 (+ 827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:07:20.297605 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:07:20.297709 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:07:20.300778 (+ 3069us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56274: BlockingRecv error: recv got EOF from 127.0.0.1:56274 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
Jun 20 06:07:20 dist-test-slave-9hm5 krb5kdc[19139](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935640, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.11.102.62@KRBTEST.COM
I20260620 06:07:20.309237 19233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:07:20.302144 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56276 (local address 127.11.102.62:36245)
0620 06:07:20.302244 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0620 06:07:20.302247 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 06:07:20.302345 (+ 98us) server_negotiation.cc:407] Connection header received
0620 06:07:20.302423 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:07:20.302425 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:07:20.302456 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:07:20.302522 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:07:20.303036 (+ 514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:07:20.303480 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:07:20.304260 (+ 780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:07:20.304445 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:07:20.304518 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:07:20.306604 (+ 2086us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:07:20.306607 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:07:20.306609 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:07:20.306628 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:07:20.307991 (+ 1363us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:07:20.308338 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:07:20.308341 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:07:20.308342 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:07:20.308412 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:07:20.308634 (+ 222us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:07:20.308636 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:07:20.308637 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:07:20.308782 (+ 145us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:07:20.308882 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0620 06:07:20.309058 (+ 176us) server_negotiation.cc:299] Negotiation successful
0620 06:07:20.309136 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260620 06:07:20.311889 19183 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56276:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 06:07:20.314765 19183 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 06:07:20.329377 19693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:07:20.323683 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44116 (local address 127.11.102.3:37845)
0620 06:07:20.324289 (+ 606us) server_negotiation.cc:206] Beginning negotiation
0620 06:07:20.324306 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0620 06:07:20.324330 (+ 24us) server_negotiation.cc:407] Connection header received
0620 06:07:20.324412 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:07:20.324418 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:07:20.324714 (+ 296us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:07:20.325208 (+ 494us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 06:07:20.325349 (+ 141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:07:20.326412 (+ 1063us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:07:20.328130 (+ 1718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:07:20.328708 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:07:20.328836 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:07:20.329000 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0620 06:07:20.329133 (+ 133us) server_negotiation.cc:299] Negotiation successful
0620 06:07:20.329225 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":517,"thread_start_us":125,"threads_started":1}
I20260620 06:07:20.329362 19689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:07:20.322892 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.102.3:37845 (local address 127.0.0.1:44116)
0620 06:07:20.323242 (+ 350us) negotiation.cc:107] Waiting for socket to connect
0620 06:07:20.323296 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0620 06:07:20.323410 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:07:20.324870 (+ 1460us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:07:20.324873 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:07:20.324891 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 06:07:20.325046 (+ 155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:07:20.325054 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:07:20.326524 (+ 1470us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:07:20.326527 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:07:20.327986 (+ 1459us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:07:20.327996 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:07:20.328833 (+ 837us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:07:20.328837 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:07:20.328942 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:07:20.328977 (+ 35us) client_negotiation.cc:769] Sending connection context
0620 06:07:20.329104 (+ 127us) client_negotiation.cc:240] Negotiation successful
0620 06:07:20.329222 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":171,"mutex_wait_us":30,"thread_start_us":85,"threads_started":1}
I20260620 06:07:20.329774 19690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:07:20.323196 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.102.1:33553 (local address 127.0.0.1:52890)
0620 06:07:20.323933 (+ 737us) negotiation.cc:107] Waiting for socket to connect
0620 06:07:20.323949 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 06:07:20.324046 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:07:20.324799 (+ 753us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:07:20.324805 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:07:20.324826 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 06:07:20.324972 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:07:20.324978 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:07:20.326424 (+ 1446us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:07:20.326426 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:07:20.328102 (+ 1676us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:07:20.328111 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:07:20.329361 (+ 1250us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:07:20.329367 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:07:20.329455 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:07:20.329482 (+ 27us) client_negotiation.cc:769] Sending connection context
0620 06:07:20.329557 (+ 75us) client_negotiation.cc:240] Negotiation successful
0620 06:07:20.329626 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":662,"mutex_wait_us":101,"thread_start_us":96,"threads_started":1}
I20260620 06:07:20.330392 19692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:07:20.323723 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52890 (local address 127.11.102.1:33553)
0620 06:07:20.324289 (+ 566us) server_negotiation.cc:206] Beginning negotiation
0620 06:07:20.324306 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0620 06:07:20.324330 (+ 24us) server_negotiation.cc:407] Connection header received
0620 06:07:20.324425 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:07:20.324430 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:07:20.324629 (+ 199us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:07:20.324814 (+ 185us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 06:07:20.325081 (+ 267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:07:20.326302 (+ 1221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:07:20.328214 (+ 1912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:07:20.329024 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:07:20.329137 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:07:20.329296 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0620 06:07:20.330109 (+ 813us) server_negotiation.cc:299] Negotiation successful
0620 06:07:20.330224 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":454,"thread_start_us":77,"threads_started":1}
I20260620 06:07:20.330997 19691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:07:20.322944 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.102.2:38643 (local address 127.0.0.1:57384)
0620 06:07:20.323826 (+ 882us) negotiation.cc:107] Waiting for socket to connect
0620 06:07:20.323839 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 06:07:20.323917 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:07:20.324987 (+ 1070us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:07:20.324989 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:07:20.325005 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 06:07:20.326572 (+ 1567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:07:20.326578 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:07:20.328055 (+ 1477us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:07:20.328061 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:07:20.329257 (+ 1196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:07:20.329269 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:07:20.330567 (+ 1298us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:07:20.330571 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:07:20.330672 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:07:20.330701 (+ 29us) client_negotiation.cc:769] Sending connection context
0620 06:07:20.330788 (+ 87us) client_negotiation.cc:240] Negotiation successful
0620 06:07:20.330860 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":816,"mutex_wait_us":1449,"thread_start_us":85,"threads_started":1}
I20260620 06:07:20.331035 19694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:07:20.324055 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57384 (local address 127.11.102.2:38643)
0620 06:07:20.324511 (+ 456us) server_negotiation.cc:206] Beginning negotiation
0620 06:07:20.324529 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0620 06:07:20.324551 (+ 22us) server_negotiation.cc:407] Connection header received
0620 06:07:20.324623 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:07:20.324628 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:07:20.324829 (+ 201us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:07:20.324975 (+ 146us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 06:07:20.326849 (+ 1874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:07:20.327942 (+ 1093us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:07:20.329418 (+ 1476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:07:20.330213 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:07:20.330321 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:07:20.330499 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0620 06:07:20.330817 (+ 318us) server_negotiation.cc:299] Negotiation successful
0620 06:07:20.330902 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":344,"thread_start_us":137,"threads_started":1}
I20260620 06:07:20.332827 19601 tablet_service.cc:1467] Processing CreateTablet for tablet 6ca7b456ee0f4c0da043dbfe702c4529 (DEFAULT_TABLE table=test-table [id=ee2f152d1dd04fbb89d15b6954537e59]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 06:07:20.333182 19321 tablet_service.cc:1467] Processing CreateTablet for tablet 6ca7b456ee0f4c0da043dbfe702c4529 (DEFAULT_TABLE table=test-table [id=ee2f152d1dd04fbb89d15b6954537e59]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 06:07:20.333448 19461 tablet_service.cc:1467] Processing CreateTablet for tablet 6ca7b456ee0f4c0da043dbfe702c4529 (DEFAULT_TABLE table=test-table [id=ee2f152d1dd04fbb89d15b6954537e59]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 06:07:20.333918 19601 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6ca7b456ee0f4c0da043dbfe702c4529. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 06:07:20.334131 19321 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6ca7b456ee0f4c0da043dbfe702c4529. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 06:07:20.334448 19461 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6ca7b456ee0f4c0da043dbfe702c4529. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 06:07:20.341253 19695 tablet_bootstrap.cc:492] T 6ca7b456ee0f4c0da043dbfe702c4529 P 4fe4e45f6e02471594ff656d8aff63ac: Bootstrap starting.
I20260620 06:07:20.342360 19697 tablet_bootstrap.cc:492] T 6ca7b456ee0f4c0da043dbfe702c4529 P 2316ce59fd744c01a82cb798c635b913: Bootstrap starting.
I20260620 06:07:20.343429 19695 tablet_bootstrap.cc:654] T 6ca7b456ee0f4c0da043dbfe702c4529 P 4fe4e45f6e02471594ff656d8aff63ac: Neither blocks nor log segments found. Creating new log.
I20260620 06:07:20.343918 19696 tablet_bootstrap.cc:492] T 6ca7b456ee0f4c0da043dbfe702c4529 P 7fe86b816edb423498c5dd937c81279c: Bootstrap starting.
I20260620 06:07:20.344168 19697 tablet_bootstrap.cc:654] T 6ca7b456ee0f4c0da043dbfe702c4529 P 2316ce59fd744c01a82cb798c635b913: Neither blocks nor log segments found. Creating new log.
I20260620 06:07:20.344352 19695 log.cc:824] T 6ca7b456ee0f4c0da043dbfe702c4529 P 4fe4e45f6e02471594ff656d8aff63ac: Log is configured to *not* fsync() on all Append() calls
I20260620 06:07:20.345077 19697 log.cc:824] T 6ca7b456ee0f4c0da043dbfe702c4529 P 2316ce59fd744c01a82cb798c635b913: Log is configured to *not* fsync() on all Append() calls
I20260620 06:07:20.345841 19696 tablet_bootstrap.cc:654] T 6ca7b456ee0f4c0da043dbfe702c4529 P 7fe86b816edb423498c5dd937c81279c: Neither blocks nor log segments found. Creating new log.
I20260620 06:07:20.346606 19696 log.cc:824] T 6ca7b456ee0f4c0da043dbfe702c4529 P 7fe86b816edb423498c5dd937c81279c: Log is configured to *not* fsync() on all Append() calls
I20260620 06:07:20.346982 19697 tablet_bootstrap.cc:492] T 6ca7b456ee0f4c0da043dbfe702c4529 P 2316ce59fd744c01a82cb798c635b913: No bootstrap required, opened a new log
I20260620 06:07:20.347007 19695 tablet_bootstrap.cc:492] T 6ca7b456ee0f4c0da043dbfe702c4529 P 4fe4e45f6e02471594ff656d8aff63ac: No bootstrap required, opened a new log
I20260620 06:07:20.347195 19695 ts_tablet_manager.cc:1397] T 6ca7b456ee0f4c0da043dbfe702c4529 P 4fe4e45f6e02471594ff656d8aff63ac: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.003s
I20260620 06:07:20.347191 19697 ts_tablet_manager.cc:1397] T 6ca7b456ee0f4c0da043dbfe702c4529 P 2316ce59fd744c01a82cb798c635b913: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260620 06:07:20.348506 19696 tablet_bootstrap.cc:492] T 6ca7b456ee0f4c0da043dbfe702c4529 P 7fe86b816edb423498c5dd937c81279c: No bootstrap required, opened a new log
I20260620 06:07:20.348667 19696 ts_tablet_manager.cc:1397] T 6ca7b456ee0f4c0da043dbfe702c4529 P 7fe86b816edb423498c5dd937c81279c: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260620 06:07:20.350179 19697 raft_consensus.cc:348] T 6ca7b456ee0f4c0da043dbfe702c4529 P 2316ce59fd744c01a82cb798c635b913 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2316ce59fd744c01a82cb798c635b913" member_type: VOTER last_known_addr { host: "127.11.102.3" port: 37845 } } peers { permanent_uuid: "4fe4e45f6e02471594ff656d8aff63ac" member_type: VOTER last_known_addr { host: "127.11.102.2" port: 38643 } } peers { permanent_uuid: "7fe86b816edb423498c5dd937c81279c" member_type: VOTER last_known_addr { host: "127.11.102.1" port: 33553 } }
I20260620 06:07:20.350410 19697 raft_consensus.cc:374] T 6ca7b456ee0f4c0da043dbfe702c4529 P 2316ce59fd744c01a82cb798c635b913 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 06:07:20.350493 19697 raft_consensus.cc:729] T 6ca7b456ee0f4c0da043dbfe702c4529 P 2316ce59fd744c01a82cb798c635b913 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2316ce59fd744c01a82cb798c635b913, State: Initialized, Role: FOLLOWER
I20260620 06:07:20.350925 19697 consensus_queue.cc:260] T 6ca7b456ee0f4c0da043dbfe702c4529 P 2316ce59fd744c01a82cb798c635b913 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2316ce59fd744c01a82cb798c635b913" member_type: VOTER last_known_addr { host: "127.11.102.3" port: 37845 } } peers { permanent_uuid: "4fe4e45f6e02471594ff656d8aff63ac" member_type: VOTER last_known_addr { host: "127.11.102.2" port: 38643 } } peers { permanent_uuid: "7fe86b816edb423498c5dd937c81279c" member_type: VOTER last_known_addr { host: "127.11.102.1" port: 33553 } }
I20260620 06:07:20.351001 19695 raft_consensus.cc:348] T 6ca7b456ee0f4c0da043dbfe702c4529 P 4fe4e45f6e02471594ff656d8aff63ac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2316ce59fd744c01a82cb798c635b913" member_type: VOTER last_known_addr { host: "127.11.102.3" port: 37845 } } peers { permanent_uuid: "4fe4e45f6e02471594ff656d8aff63ac" member_type: VOTER last_known_addr { host: "127.11.102.2" port: 38643 } } peers { permanent_uuid: "7fe86b816edb423498c5dd937c81279c" member_type: VOTER last_known_addr { host: "127.11.102.1" port: 33553 } }
I20260620 06:07:20.351305 19695 raft_consensus.cc:374] T 6ca7b456ee0f4c0da043dbfe702c4529 P 4fe4e45f6e02471594ff656d8aff63ac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 06:07:20.351397 19695 raft_consensus.cc:729] T 6ca7b456ee0f4c0da043dbfe702c4529 P 4fe4e45f6e02471594ff656d8aff63ac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4fe4e45f6e02471594ff656d8aff63ac, State: Initialized, Role: FOLLOWER
I20260620 06:07:20.351881 19695 consensus_queue.cc:260] T 6ca7b456ee0f4c0da043dbfe702c4529 P 4fe4e45f6e02471594ff656d8aff63ac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2316ce59fd744c01a82cb798c635b913" member_type: VOTER last_known_addr { host: "127.11.102.3" port: 37845 } } peers { permanent_uuid: "4fe4e45f6e02471594ff656d8aff63ac" member_type: VOTER last_known_addr { host: "127.11.102.2" port: 38643 } } peers { permanent_uuid: "7fe86b816edb423498c5dd937c81279c" member_type: VOTER last_known_addr { host: "127.11.102.1" port: 33553 } }
I20260620 06:07:20.352043 19697 ts_tablet_manager.cc:1428] T 6ca7b456ee0f4c0da043dbfe702c4529 P 2316ce59fd744c01a82cb798c635b913: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260620 06:07:20.352404 19667 heartbeater.cc:503] Master 127.11.102.62:36245 was elected leader, sending a full tablet report...
I20260620 06:07:20.352470 19527 heartbeater.cc:503] Master 127.11.102.62:36245 was elected leader, sending a full tablet report...
I20260620 06:07:20.352777 19695 ts_tablet_manager.cc:1428] T 6ca7b456ee0f4c0da043dbfe702c4529 P 4fe4e45f6e02471594ff656d8aff63ac: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260620 06:07:20.353295 19696 raft_consensus.cc:348] T 6ca7b456ee0f4c0da043dbfe702c4529 P 7fe86b816edb423498c5dd937c81279c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2316ce59fd744c01a82cb798c635b913" member_type: VOTER last_known_addr { host: "127.11.102.3" port: 37845 } } peers { permanent_uuid: "4fe4e45f6e02471594ff656d8aff63ac" member_type: VOTER last_known_addr { host: "127.11.102.2" port: 38643 } } peers { permanent_uuid: "7fe86b816edb423498c5dd937c81279c" member_type: VOTER last_known_addr { host: "127.11.102.1" port: 33553 } }
I20260620 06:07:20.353603 19696 raft_consensus.cc:374] T 6ca7b456ee0f4c0da043dbfe702c4529 P 7fe86b816edb423498c5dd937c81279c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 06:07:20.353688 19696 raft_consensus.cc:729] T 6ca7b456ee0f4c0da043dbfe702c4529 P 7fe86b816edb423498c5dd937c81279c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7fe86b816edb423498c5dd937c81279c, State: Initialized, Role: FOLLOWER
I20260620 06:07:20.354179 19696 consensus_queue.cc:260] T 6ca7b456ee0f4c0da043dbfe702c4529 P 7fe86b816edb423498c5dd937c81279c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2316ce59fd744c01a82cb798c635b913" member_type: VOTER last_known_addr { host: "127.11.102.3" port: 37845 } } peers { permanent_uuid: "4fe4e45f6e02471594ff656d8aff63ac" member_type: VOTER last_known_addr { host: "127.11.102.2" port: 38643 } } peers { permanent_uuid: "7fe86b816edb423498c5dd937c81279c" member_type: VOTER last_known_addr { host: "127.11.102.1" port: 33553 } }
I20260620 06:07:20.354796 19387 heartbeater.cc:503] Master 127.11.102.62:36245 was elected leader, sending a full tablet report...
I20260620 06:07:20.355104 19696 ts_tablet_manager.cc:1428] T 6ca7b456ee0f4c0da043dbfe702c4529 P 7fe86b816edb423498c5dd937c81279c: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260620 06:07:20.402434 19701 raft_consensus.cc:482] T 6ca7b456ee0f4c0da043dbfe702c4529 P 2316ce59fd744c01a82cb798c635b913 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 06:07:20.402624 19701 raft_consensus.cc:504] T 6ca7b456ee0f4c0da043dbfe702c4529 P 2316ce59fd744c01a82cb798c635b913 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2316ce59fd744c01a82cb798c635b913" member_type: VOTER last_known_addr { host: "127.11.102.3" port: 37845 } } peers { permanent_uuid: "4fe4e45f6e02471594ff656d8aff63ac" member_type: VOTER last_known_addr { host: "127.11.102.2" port: 38643 } } peers { permanent_uuid: "7fe86b816edb423498c5dd937c81279c" member_type: VOTER last_known_addr { host: "127.11.102.1" port: 33553 } }
I20260620 06:07:20.403757 19701 leader_election.cc:288] T 6ca7b456ee0f4c0da043dbfe702c4529 P 2316ce59fd744c01a82cb798c635b913 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4fe4e45f6e02471594ff656d8aff63ac (127.11.102.2:38643), 7fe86b816edb423498c5dd937c81279c (127.11.102.1:33553)
I20260620 06:07:20.407444 19560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:07:20.404008 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.102.1:33553 (local address 127.11.102.3:33043)
0620 06:07:20.404125 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0620 06:07:20.404142 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 06:07:20.404235 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:07:20.404489 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:07:20.404492 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:07:20.404511 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 06:07:20.404650 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:07:20.404657 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:07:20.405459 (+ 802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:07:20.405462 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:07:20.406457 (+ 995us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:07:20.406467 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:07:20.407135 (+ 668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:07:20.407137 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:07:20.407213 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:07:20.407231 (+ 18us) client_negotiation.cc:769] Sending connection context
0620 06:07:20.407288 (+ 57us) client_negotiation.cc:240] Negotiation successful
0620 06:07:20.407343 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33}
I20260620 06:07:20.407524 19692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:07:20.404104 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.102.3:33043 (local address 127.11.102.1:33553)
0620 06:07:20.404239 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0620 06:07:20.404245 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 06:07:20.404266 (+ 21us) server_negotiation.cc:407] Connection header received
0620 06:07:20.404325 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:07:20.404328 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:07:20.404376 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:07:20.404467 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 06:07:20.404782 (+ 315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:07:20.405356 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:07:20.406606 (+ 1250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:07:20.407017 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:07:20.407151 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:07:20.407204 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0620 06:07:20.407329 (+ 125us) server_negotiation.cc:299] Negotiation successful
0620 06:07:20.407403 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260620 06:07:20.408165 19341 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ca7b456ee0f4c0da043dbfe702c4529" candidate_uuid: "2316ce59fd744c01a82cb798c635b913" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fe86b816edb423498c5dd937c81279c" is_pre_election: true
I20260620 06:07:20.408222 19704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:07:20.404008 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.102.2:38643 (local address 127.11.102.3:59959)
0620 06:07:20.404493 (+ 485us) negotiation.cc:107] Waiting for socket to connect
0620 06:07:20.404505 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 06:07:20.404587 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:07:20.404850 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:07:20.404852 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:07:20.404866 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 06:07:20.405286 (+ 420us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:07:20.405291 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:07:20.406020 (+ 729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:07:20.406023 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:07:20.406915 (+ 892us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:07:20.406922 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:07:20.407815 (+ 893us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:07:20.407819 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:07:20.407924 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:07:20.407939 (+ 15us) client_negotiation.cc:769] Sending connection context
0620 06:07:20.408027 (+ 88us) client_negotiation.cc:240] Negotiation successful
0620 06:07:20.408089 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":399,"mutex_wait_us":168,"thread_start_us":101,"threads_started":1}
I20260620 06:07:20.408218 19694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:07:20.404126 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.102.3:59959 (local address 127.11.102.2:38643)
0620 06:07:20.404238 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0620 06:07:20.404243 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 06:07:20.404568 (+ 325us) server_negotiation.cc:407] Connection header received
0620 06:07:20.404682 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:07:20.404684 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:07:20.404749 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:07:20.404830 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 06:07:20.405404 (+ 574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:07:20.405905 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:07:20.407024 (+ 1119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:07:20.407662 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:07:20.407817 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:07:20.407861 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0620 06:07:20.408055 (+ 194us) server_negotiation.cc:299] Negotiation successful
0620 06:07:20.408118 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260620 06:07:20.408502 19341 raft_consensus.cc:2454] T 6ca7b456ee0f4c0da043dbfe702c4529 P 7fe86b816edb423498c5dd937c81279c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2316ce59fd744c01a82cb798c635b913 in term 0.
I20260620 06:07:20.408767 19471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ca7b456ee0f4c0da043dbfe702c4529" candidate_uuid: "2316ce59fd744c01a82cb798c635b913" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4fe4e45f6e02471594ff656d8aff63ac" is_pre_election: true
I20260620 06:07:20.409061 19471 raft_consensus.cc:2454] T 6ca7b456ee0f4c0da043dbfe702c4529 P 4fe4e45f6e02471594ff656d8aff63ac [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2316ce59fd744c01a82cb798c635b913 in term 0.
I20260620 06:07:20.409029 19553 leader_election.cc:302] T 6ca7b456ee0f4c0da043dbfe702c4529 P 2316ce59fd744c01a82cb798c635b913 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2316ce59fd744c01a82cb798c635b913, 7fe86b816edb423498c5dd937c81279c; no voters:
I20260620 06:07:20.409338 19701 raft_consensus.cc:2781] T 6ca7b456ee0f4c0da043dbfe702c4529 P 2316ce59fd744c01a82cb798c635b913 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 06:07:20.409435 19701 raft_consensus.cc:482] T 6ca7b456ee0f4c0da043dbfe702c4529 P 2316ce59fd744c01a82cb798c635b913 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 06:07:20.409514 19701 raft_consensus.cc:3037] T 6ca7b456ee0f4c0da043dbfe702c4529 P 2316ce59fd744c01a82cb798c635b913 [term 0 FOLLOWER]: Advancing to term 1
I20260620 06:07:20.410627 19701 raft_consensus.cc:504] T 6ca7b456ee0f4c0da043dbfe702c4529 P 2316ce59fd744c01a82cb798c635b913 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2316ce59fd744c01a82cb798c635b913" member_type: VOTER last_known_addr { host: "127.11.102.3" port: 37845 } } peers { permanent_uuid: "4fe4e45f6e02471594ff656d8aff63ac" member_type: VOTER last_known_addr { host: "127.11.102.2" port: 38643 } } peers { permanent_uuid: "7fe86b816edb423498c5dd937c81279c" member_type: VOTER last_known_addr { host: "127.11.102.1" port: 33553 } }
I20260620 06:07:20.410975 19701 leader_election.cc:288] T 6ca7b456ee0f4c0da043dbfe702c4529 P 2316ce59fd744c01a82cb798c635b913 [CANDIDATE]: Term 1 election: Requested vote from peers 4fe4e45f6e02471594ff656d8aff63ac (127.11.102.2:38643), 7fe86b816edb423498c5dd937c81279c (127.11.102.1:33553)
I20260620 06:07:20.411348 19471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ca7b456ee0f4c0da043dbfe702c4529" candidate_uuid: "2316ce59fd744c01a82cb798c635b913" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4fe4e45f6e02471594ff656d8aff63ac"
I20260620 06:07:20.411490 19471 raft_consensus.cc:3037] T 6ca7b456ee0f4c0da043dbfe702c4529 P 4fe4e45f6e02471594ff656d8aff63ac [term 0 FOLLOWER]: Advancing to term 1
I20260620 06:07:20.411454 19341 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ca7b456ee0f4c0da043dbfe702c4529" candidate_uuid: "2316ce59fd744c01a82cb798c635b913" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fe86b816edb423498c5dd937c81279c"
I20260620 06:07:20.411672 19341 raft_consensus.cc:3037] T 6ca7b456ee0f4c0da043dbfe702c4529 P 7fe86b816edb423498c5dd937c81279c [term 0 FOLLOWER]: Advancing to term 1
I20260620 06:07:20.412576 19471 raft_consensus.cc:2454] T 6ca7b456ee0f4c0da043dbfe702c4529 P 4fe4e45f6e02471594ff656d8aff63ac [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2316ce59fd744c01a82cb798c635b913 in term 1.
I20260620 06:07:20.412629 19341 raft_consensus.cc:2454] T 6ca7b456ee0f4c0da043dbfe702c4529 P 7fe86b816edb423498c5dd937c81279c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2316ce59fd744c01a82cb798c635b913 in term 1.
I20260620 06:07:20.412858 19555 leader_election.cc:302] T 6ca7b456ee0f4c0da043dbfe702c4529 P 2316ce59fd744c01a82cb798c635b913 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2316ce59fd744c01a82cb798c635b913, 4fe4e45f6e02471594ff656d8aff63ac; no voters:
I20260620 06:07:20.413051 19701 raft_consensus.cc:2781] T 6ca7b456ee0f4c0da043dbfe702c4529 P 2316ce59fd744c01a82cb798c635b913 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 06:07:20.413262 19701 raft_consensus.cc:686] T 6ca7b456ee0f4c0da043dbfe702c4529 P 2316ce59fd744c01a82cb798c635b913 [term 1 LEADER]: Becoming Leader. State: Replica: 2316ce59fd744c01a82cb798c635b913, State: Running, Role: LEADER
I20260620 06:07:20.413605 19701 consensus_queue.cc:237] T 6ca7b456ee0f4c0da043dbfe702c4529 P 2316ce59fd744c01a82cb798c635b913 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2316ce59fd744c01a82cb798c635b913" member_type: VOTER last_known_addr { host: "127.11.102.3" port: 37845 } } peers { permanent_uuid: "4fe4e45f6e02471594ff656d8aff63ac" member_type: VOTER last_known_addr { host: "127.11.102.2" port: 38643 } } peers { permanent_uuid: "7fe86b816edb423498c5dd937c81279c" member_type: VOTER last_known_addr { host: "127.11.102.1" port: 33553 } }
I20260620 06:07:20.416891 19182 catalog_manager.cc:5495] T 6ca7b456ee0f4c0da043dbfe702c4529 P 2316ce59fd744c01a82cb798c635b913 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2316ce59fd744c01a82cb798c635b913 (127.11.102.3). New cstate: current_term: 1 leader_uuid: "2316ce59fd744c01a82cb798c635b913" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2316ce59fd744c01a82cb798c635b913" member_type: VOTER last_known_addr { host: "127.11.102.3" port: 37845 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4fe4e45f6e02471594ff656d8aff63ac" member_type: VOTER last_known_addr { host: "127.11.102.2" port: 38643 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7fe86b816edb423498c5dd937c81279c" member_type: VOTER last_known_addr { host: "127.11.102.1" port: 33553 } health_report { overall_health: UNKNOWN } } }
I20260620 06:07:20.449772 19693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:07:20.446200 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44126 (local address 127.11.102.3:37845)
0620 06:07:20.446338 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0620 06:07:20.446343 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 06:07:20.446357 (+ 14us) server_negotiation.cc:407] Connection header received
0620 06:07:20.446528 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:07:20.446530 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:07:20.446583 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:07:20.446698 (+ 115us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 06:07:20.446926 (+ 228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:07:20.447449 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:07:20.448346 (+ 897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:07:20.448580 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:07:20.448695 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:07:20.448964 (+ 269us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 06:07:20.449381 (+ 417us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 06:07:20.449512 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0620 06:07:20.449576 (+ 64us) server_negotiation.cc:299] Negotiation successful
0620 06:07:20.449629 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260620 06:07:20.455348 19341 raft_consensus.cc:1261] T 6ca7b456ee0f4c0da043dbfe702c4529 P 7fe86b816edb423498c5dd937c81279c [term 1 FOLLOWER]: Refusing update from remote peer 2316ce59fd744c01a82cb798c635b913: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 06:07:20.455397 19471 raft_consensus.cc:1261] T 6ca7b456ee0f4c0da043dbfe702c4529 P 4fe4e45f6e02471594ff656d8aff63ac [term 1 FOLLOWER]: Refusing update from remote peer 2316ce59fd744c01a82cb798c635b913: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 06:07:20.456095 19701 consensus_queue.cc:1035] T 6ca7b456ee0f4c0da043dbfe702c4529 P 2316ce59fd744c01a82cb798c635b913 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7fe86b816edb423498c5dd937c81279c" member_type: VOTER last_known_addr { host: "127.11.102.1" port: 33553 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 06:07:20.456317 19705 consensus_queue.cc:1035] T 6ca7b456ee0f4c0da043dbfe702c4529 P 2316ce59fd744c01a82cb798c635b913 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4fe4e45f6e02471594ff656d8aff63ac" member_type: VOTER last_known_addr { host: "127.11.102.2" port: 38643 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 06:07:20.463263 19708 mvcc.cc:205] Tried to move back new op lower bound from 7298808383298777088 to 7298808383137439744. Current Snapshot: MvccSnapshot[applied={T|T < 7298808383298777088}]
I20260620 06:07:20.465054 19709 mvcc.cc:205] Tried to move back new op lower bound from 7298808383298777088 to 7298808383137439744. Current Snapshot: MvccSnapshot[applied={T|T < 7298808383298777088}]
I20260620 06:07:20.466527 19710 mvcc.cc:205] Tried to move back new op lower bound from 7298808383298777088 to 7298808383137439744. Current Snapshot: MvccSnapshot[applied={T|T < 7298808383298777088}]
I20260620 06:07:20.471259 19182 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56276:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 06:07:20.471436 19182 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56276:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 06:07:20.474222 19182 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 05824e2e98474e8da81f776cc916c2e8: Sending DeleteTablet for 3 replicas of tablet 6ca7b456ee0f4c0da043dbfe702c4529
I20260620 06:07:20.474987 19321 tablet_service.cc:1514] Processing DeleteTablet for tablet 6ca7b456ee0f4c0da043dbfe702c4529 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 06:07:20 UTC) from {username='oryx', principal='oryx/127.11.102.62@KRBTEST.COM'} at 127.0.0.1:52890
I20260620 06:07:20.474984 19601 tablet_service.cc:1514] Processing DeleteTablet for tablet 6ca7b456ee0f4c0da043dbfe702c4529 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 06:07:20 UTC) from {username='oryx', principal='oryx/127.11.102.62@KRBTEST.COM'} at 127.0.0.1:44116
I20260620 06:07:20.475457 19718 tablet_replica.cc:330] T 6ca7b456ee0f4c0da043dbfe702c4529 P 7fe86b816edb423498c5dd937c81279c: stopping tablet replica
I20260620 06:07:20.475466 19461 tablet_service.cc:1514] Processing DeleteTablet for tablet 6ca7b456ee0f4c0da043dbfe702c4529 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 06:07:20 UTC) from {username='oryx', principal='oryx/127.11.102.62@KRBTEST.COM'} at 127.0.0.1:57384
I20260620 06:07:20.475785 19718 raft_consensus.cc:2229] T 6ca7b456ee0f4c0da043dbfe702c4529 P 7fe86b816edb423498c5dd937c81279c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 06:07:20.476069 19719 tablet_replica.cc:330] T 6ca7b456ee0f4c0da043dbfe702c4529 P 4fe4e45f6e02471594ff656d8aff63ac: stopping tablet replica
I20260620 06:07:20.476106 19718 raft_consensus.cc:2258] T 6ca7b456ee0f4c0da043dbfe702c4529 P 7fe86b816edb423498c5dd937c81279c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 06:07:20.476447 19717 tablet_replica.cc:330] T 6ca7b456ee0f4c0da043dbfe702c4529 P 2316ce59fd744c01a82cb798c635b913: stopping tablet replica
I20260620 06:07:20.476449 19719 raft_consensus.cc:2229] T 6ca7b456ee0f4c0da043dbfe702c4529 P 4fe4e45f6e02471594ff656d8aff63ac [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 06:07:20.476711 19717 raft_consensus.cc:2229] T 6ca7b456ee0f4c0da043dbfe702c4529 P 2316ce59fd744c01a82cb798c635b913 [term 1 LEADER]: Raft consensus shutting down.
I20260620 06:07:20.476841 19719 raft_consensus.cc:2258] T 6ca7b456ee0f4c0da043dbfe702c4529 P 4fe4e45f6e02471594ff656d8aff63ac [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 06:07:20.477084 19717 raft_consensus.cc:2258] T 6ca7b456ee0f4c0da043dbfe702c4529 P 2316ce59fd744c01a82cb798c635b913 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 06:07:20.477012 19184 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56276:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 06:07:20.477454 19184 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 06:07:20.478590 19719 ts_tablet_manager.cc:1905] T 6ca7b456ee0f4c0da043dbfe702c4529 P 4fe4e45f6e02471594ff656d8aff63ac: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 06:07:20.479875 19717 ts_tablet_manager.cc:1905] T 6ca7b456ee0f4c0da043dbfe702c4529 P 2316ce59fd744c01a82cb798c635b913: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 06:07:20.480614 19718 ts_tablet_manager.cc:1905] T 6ca7b456ee0f4c0da043dbfe702c4529 P 7fe86b816edb423498c5dd937c81279c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 06:07:20.482692 19717 ts_tablet_manager.cc:1918] T 6ca7b456ee0f4c0da043dbfe702c4529 P 2316ce59fd744c01a82cb798c635b913: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 06:07:20.482838 19717 log.cc:1196] T 6ca7b456ee0f4c0da043dbfe702c4529 P 2316ce59fd744c01a82cb798c635b913: Deleting WAL directory at /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781935603120653-11672-0/minicluster-data/ts-2/wal/wals/6ca7b456ee0f4c0da043dbfe702c4529
I20260620 06:07:20.483202 19718 ts_tablet_manager.cc:1918] T 6ca7b456ee0f4c0da043dbfe702c4529 P 7fe86b816edb423498c5dd937c81279c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 06:07:20.483297 19718 log.cc:1196] T 6ca7b456ee0f4c0da043dbfe702c4529 P 7fe86b816edb423498c5dd937c81279c: Deleting WAL directory at /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781935603120653-11672-0/minicluster-data/ts-0/wal/wals/6ca7b456ee0f4c0da043dbfe702c4529
I20260620 06:07:20.483603 19718 ts_tablet_manager.cc:1939] T 6ca7b456ee0f4c0da043dbfe702c4529 P 7fe86b816edb423498c5dd937c81279c: Deleting consensus metadata
I20260620 06:07:20.483726 19461 tablet_service.cc:1467] Processing CreateTablet for tablet 09d9ebc901d1409db79307028aa69ffa (DEFAULT_TABLE table=test-table [id=c95c167070e944d8b5faf77ed80b5c98]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 06:07:20.483983 19461 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 09d9ebc901d1409db79307028aa69ffa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 06:07:20.484087 19601 tablet_service.cc:1467] Processing CreateTablet for tablet 09d9ebc901d1409db79307028aa69ffa (DEFAULT_TABLE table=test-table [id=c95c167070e944d8b5faf77ed80b5c98]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 06:07:20.484324 19601 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 09d9ebc901d1409db79307028aa69ffa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 06:07:20.484865 19169 catalog_manager.cc:4841] TS 7fe86b816edb423498c5dd937c81279c (127.11.102.1:33553): tablet 6ca7b456ee0f4c0da043dbfe702c4529 (table test-table [id=ee2f152d1dd04fbb89d15b6954537e59]) successfully deleted
I20260620 06:07:20.485044 19717 ts_tablet_manager.cc:1939] T 6ca7b456ee0f4c0da043dbfe702c4529 P 2316ce59fd744c01a82cb798c635b913: Deleting consensus metadata
I20260620 06:07:20.486123 19170 catalog_manager.cc:4841] TS 2316ce59fd744c01a82cb798c635b913 (127.11.102.3:37845): tablet 6ca7b456ee0f4c0da043dbfe702c4529 (table test-table [id=ee2f152d1dd04fbb89d15b6954537e59]) successfully deleted
I20260620 06:07:20.486366 19697 tablet_bootstrap.cc:492] T 09d9ebc901d1409db79307028aa69ffa P 2316ce59fd744c01a82cb798c635b913: Bootstrap starting.
I20260620 06:07:20.486552 19321 tablet_service.cc:1467] Processing CreateTablet for tablet 09d9ebc901d1409db79307028aa69ffa (DEFAULT_TABLE table=test-table [id=c95c167070e944d8b5faf77ed80b5c98]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 06:07:20.486791 19321 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 09d9ebc901d1409db79307028aa69ffa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 06:07:20.487303 19697 tablet_bootstrap.cc:654] T 09d9ebc901d1409db79307028aa69ffa P 2316ce59fd744c01a82cb798c635b913: Neither blocks nor log segments found. Creating new log.
I20260620 06:07:20.488806 19697 tablet_bootstrap.cc:492] T 09d9ebc901d1409db79307028aa69ffa P 2316ce59fd744c01a82cb798c635b913: No bootstrap required, opened a new log
I20260620 06:07:20.488888 19697 ts_tablet_manager.cc:1397] T 09d9ebc901d1409db79307028aa69ffa P 2316ce59fd744c01a82cb798c635b913: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20260620 06:07:20.489311 19697 raft_consensus.cc:348] T 09d9ebc901d1409db79307028aa69ffa P 2316ce59fd744c01a82cb798c635b913 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fe86b816edb423498c5dd937c81279c" member_type: VOTER last_known_addr { host: "127.11.102.1" port: 33553 } } peers { permanent_uuid: "4fe4e45f6e02471594ff656d8aff63ac" member_type: VOTER last_known_addr { host: "127.11.102.2" port: 38643 } } peers { permanent_uuid: "2316ce59fd744c01a82cb798c635b913" member_type: VOTER last_known_addr { host: "127.11.102.3" port: 37845 } }
I20260620 06:07:20.489418 19697 raft_consensus.cc:374] T 09d9ebc901d1409db79307028aa69ffa P 2316ce59fd744c01a82cb798c635b913 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 06:07:20.489455 19697 raft_consensus.cc:729] T 09d9ebc901d1409db79307028aa69ffa P 2316ce59fd744c01a82cb798c635b913 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2316ce59fd744c01a82cb798c635b913, State: Initialized, Role: FOLLOWER
I20260620 06:07:20.489704 19695 tablet_bootstrap.cc:492] T 09d9ebc901d1409db79307028aa69ffa P 4fe4e45f6e02471594ff656d8aff63ac: Bootstrap starting.
I20260620 06:07:20.489773 19696 tablet_bootstrap.cc:492] T 09d9ebc901d1409db79307028aa69ffa P 7fe86b816edb423498c5dd937c81279c: Bootstrap starting.
I20260620 06:07:20.490463 19695 tablet_bootstrap.cc:654] T 09d9ebc901d1409db79307028aa69ffa P 4fe4e45f6e02471594ff656d8aff63ac: Neither blocks nor log segments found. Creating new log.
I20260620 06:07:20.490628 19696 tablet_bootstrap.cc:654] T 09d9ebc901d1409db79307028aa69ffa P 7fe86b816edb423498c5dd937c81279c: Neither blocks nor log segments found. Creating new log.
I20260620 06:07:20.489583 19697 consensus_queue.cc:260] T 09d9ebc901d1409db79307028aa69ffa P 2316ce59fd744c01a82cb798c635b913 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fe86b816edb423498c5dd937c81279c" member_type: VOTER last_known_addr { host: "127.11.102.1" port: 33553 } } peers { permanent_uuid: "4fe4e45f6e02471594ff656d8aff63ac" member_type: VOTER last_known_addr { host: "127.11.102.2" port: 38643 } } peers { permanent_uuid: "2316ce59fd744c01a82cb798c635b913" member_type: VOTER last_known_addr { host: "127.11.102.3" port: 37845 } }
I20260620 06:07:20.491596 19695 tablet_bootstrap.cc:492] T 09d9ebc901d1409db79307028aa69ffa P 4fe4e45f6e02471594ff656d8aff63ac: No bootstrap required, opened a new log
I20260620 06:07:20.491707 19695 ts_tablet_manager.cc:1397] T 09d9ebc901d1409db79307028aa69ffa P 4fe4e45f6e02471594ff656d8aff63ac: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.001s
I20260620 06:07:20.491786 19697 ts_tablet_manager.cc:1428] T 09d9ebc901d1409db79307028aa69ffa P 2316ce59fd744c01a82cb798c635b913: Time spent starting tablet: real 0.003s user 0.000s sys 0.001s
I20260620 06:07:20.492147 19695 raft_consensus.cc:348] T 09d9ebc901d1409db79307028aa69ffa P 4fe4e45f6e02471594ff656d8aff63ac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fe86b816edb423498c5dd937c81279c" member_type: VOTER last_known_addr { host: "127.11.102.1" port: 33553 } } peers { permanent_uuid: "4fe4e45f6e02471594ff656d8aff63ac" member_type: VOTER last_known_addr { host: "127.11.102.2" port: 38643 } } peers { permanent_uuid: "2316ce59fd744c01a82cb798c635b913" member_type: VOTER last_known_addr { host: "127.11.102.3" port: 37845 } }
I20260620 06:07:20.492255 19695 raft_consensus.cc:374] T 09d9ebc901d1409db79307028aa69ffa P 4fe4e45f6e02471594ff656d8aff63ac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 06:07:20.492292 19695 raft_consensus.cc:729] T 09d9ebc901d1409db79307028aa69ffa P 4fe4e45f6e02471594ff656d8aff63ac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4fe4e45f6e02471594ff656d8aff63ac, State: Initialized, Role: FOLLOWER
I20260620 06:07:20.492414 19695 consensus_queue.cc:260] T 09d9ebc901d1409db79307028aa69ffa P 4fe4e45f6e02471594ff656d8aff63ac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fe86b816edb423498c5dd937c81279c" member_type: VOTER last_known_addr { host: "127.11.102.1" port: 33553 } } peers { permanent_uuid: "4fe4e45f6e02471594ff656d8aff63ac" member_type: VOTER last_known_addr { host: "127.11.102.2" port: 38643 } } peers { permanent_uuid: "2316ce59fd744c01a82cb798c635b913" member_type: VOTER last_known_addr { host: "127.11.102.3" port: 37845 } }
I20260620 06:07:20.492648 19695 ts_tablet_manager.cc:1428] T 09d9ebc901d1409db79307028aa69ffa P 4fe4e45f6e02471594ff656d8aff63ac: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260620 06:07:20.493111 19719 ts_tablet_manager.cc:1918] T 6ca7b456ee0f4c0da043dbfe702c4529 P 4fe4e45f6e02471594ff656d8aff63ac: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 06:07:20.493199 19719 log.cc:1196] T 6ca7b456ee0f4c0da043dbfe702c4529 P 4fe4e45f6e02471594ff656d8aff63ac: Deleting WAL directory at /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781935603120653-11672-0/minicluster-data/ts-1/wal/wals/6ca7b456ee0f4c0da043dbfe702c4529
I20260620 06:07:20.493489 19719 ts_tablet_manager.cc:1939] T 6ca7b456ee0f4c0da043dbfe702c4529 P 4fe4e45f6e02471594ff656d8aff63ac: Deleting consensus metadata
I20260620 06:07:20.494633 19171 catalog_manager.cc:4841] TS 4fe4e45f6e02471594ff656d8aff63ac (127.11.102.2:38643): tablet 6ca7b456ee0f4c0da043dbfe702c4529 (table test-table [id=ee2f152d1dd04fbb89d15b6954537e59]) successfully deleted
I20260620 06:07:20.495203 19696 tablet_bootstrap.cc:492] T 09d9ebc901d1409db79307028aa69ffa P 7fe86b816edb423498c5dd937c81279c: No bootstrap required, opened a new log
I20260620 06:07:20.495270 19696 ts_tablet_manager.cc:1397] T 09d9ebc901d1409db79307028aa69ffa P 7fe86b816edb423498c5dd937c81279c: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.000s
I20260620 06:07:20.495934 19696 raft_consensus.cc:348] T 09d9ebc901d1409db79307028aa69ffa P 7fe86b816edb423498c5dd937c81279c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fe86b816edb423498c5dd937c81279c" member_type: VOTER last_known_addr { host: "127.11.102.1" port: 33553 } } peers { permanent_uuid: "4fe4e45f6e02471594ff656d8aff63ac" member_type: VOTER last_known_addr { host: "127.11.102.2" port: 38643 } } peers { permanent_uuid: "2316ce59fd744c01a82cb798c635b913" member_type: VOTER last_known_addr { host: "127.11.102.3" port: 37845 } }
I20260620 06:07:20.496024 19696 raft_consensus.cc:374] T 09d9ebc901d1409db79307028aa69ffa P 7fe86b816edb423498c5dd937c81279c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 06:07:20.496047 19696 raft_consensus.cc:729] T 09d9ebc901d1409db79307028aa69ffa P 7fe86b816edb423498c5dd937c81279c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7fe86b816edb423498c5dd937c81279c, State: Initialized, Role: FOLLOWER
I20260620 06:07:20.496196 19696 consensus_queue.cc:260] T 09d9ebc901d1409db79307028aa69ffa P 7fe86b816edb423498c5dd937c81279c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fe86b816edb423498c5dd937c81279c" member_type: VOTER last_known_addr { host: "127.11.102.1" port: 33553 } } peers { permanent_uuid: "4fe4e45f6e02471594ff656d8aff63ac" member_type: VOTER last_known_addr { host: "127.11.102.2" port: 38643 } } peers { permanent_uuid: "2316ce59fd744c01a82cb798c635b913" member_type: VOTER last_known_addr { host: "127.11.102.3" port: 37845 } }
I20260620 06:07:20.496449 19696 ts_tablet_manager.cc:1428] T 09d9ebc901d1409db79307028aa69ffa P 7fe86b816edb423498c5dd937c81279c: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
W20260620 06:07:20.520108 19668 tablet.cc:2369] T 09d9ebc901d1409db79307028aa69ffa P 2316ce59fd744c01a82cb798c635b913: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 06:07:20.555311 19528 tablet.cc:2369] T 09d9ebc901d1409db79307028aa69ffa P 4fe4e45f6e02471594ff656d8aff63ac: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 06:07:20.585644 19388 tablet.cc:2369] T 09d9ebc901d1409db79307028aa69ffa P 7fe86b816edb423498c5dd937c81279c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 06:07:20.754659 19702 raft_consensus.cc:482] T 09d9ebc901d1409db79307028aa69ffa P 4fe4e45f6e02471594ff656d8aff63ac [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 06:07:20.754828 19702 raft_consensus.cc:504] T 09d9ebc901d1409db79307028aa69ffa P 4fe4e45f6e02471594ff656d8aff63ac [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fe86b816edb423498c5dd937c81279c" member_type: VOTER last_known_addr { host: "127.11.102.1" port: 33553 } } peers { permanent_uuid: "4fe4e45f6e02471594ff656d8aff63ac" member_type: VOTER last_known_addr { host: "127.11.102.2" port: 38643 } } peers { permanent_uuid: "2316ce59fd744c01a82cb798c635b913" member_type: VOTER last_known_addr { host: "127.11.102.3" port: 37845 } }
I20260620 06:07:20.755877 19702 leader_election.cc:288] T 09d9ebc901d1409db79307028aa69ffa P 4fe4e45f6e02471594ff656d8aff63ac [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7fe86b816edb423498c5dd937c81279c (127.11.102.1:33553), 2316ce59fd744c01a82cb798c635b913 (127.11.102.3:37845)
I20260620 06:07:20.759972 19692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:07:20.756371 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.102.2:47037 (local address 127.11.102.1:33553)
0620 06:07:20.756510 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0620 06:07:20.756514 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 06:07:20.756568 (+ 54us) server_negotiation.cc:407] Connection header received
0620 06:07:20.756775 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:07:20.756783 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:07:20.756829 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:07:20.756907 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 06:07:20.757184 (+ 277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:07:20.757772 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:07:20.758899 (+ 1127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:07:20.759315 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:07:20.759406 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:07:20.759515 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0620 06:07:20.759753 (+ 238us) server_negotiation.cc:299] Negotiation successful
0620 06:07:20.759838 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260620 06:07:20.759999 19693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:07:20.756252 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.102.2:42477 (local address 127.11.102.3:37845)
0620 06:07:20.756392 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0620 06:07:20.756396 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 06:07:20.756638 (+ 242us) server_negotiation.cc:407] Connection header received
0620 06:07:20.756778 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:07:20.756788 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:07:20.756832 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:07:20.756919 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 06:07:20.757225 (+ 306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:07:20.757706 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:07:20.758978 (+ 1272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:07:20.759420 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:07:20.759531 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:07:20.759591 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0620 06:07:20.759798 (+ 207us) server_negotiation.cc:299] Negotiation successful
0620 06:07:20.759878 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260620 06:07:20.760031 19723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:07:20.756140 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.102.1:33553 (local address 127.11.102.2:47037)
0620 06:07:20.756495 (+ 355us) negotiation.cc:107] Waiting for socket to connect
0620 06:07:20.756507 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 06:07:20.756635 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:07:20.756921 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:07:20.756924 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:07:20.756939 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 06:07:20.757075 (+ 136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:07:20.757081 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:07:20.757895 (+ 814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:07:20.757898 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:07:20.758786 (+ 888us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:07:20.758794 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:07:20.759438 (+ 644us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:07:20.759444 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:07:20.759584 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:07:20.759636 (+ 52us) client_negotiation.cc:769] Sending connection context
0620 06:07:20.759707 (+ 71us) client_negotiation.cc:240] Negotiation successful
0620 06:07:20.759812 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":281,"mutex_wait_us":26,"spinlock_wait_cycles":16640,"thread_start_us":86,"threads_started":1}
I20260620 06:07:20.760458 19724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:07:20.756140 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.102.3:37845 (local address 127.11.102.2:42477)
0620 06:07:20.756515 (+ 375us) negotiation.cc:107] Waiting for socket to connect
0620 06:07:20.756530 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 06:07:20.756651 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:07:20.756948 (+ 297us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:07:20.756951 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:07:20.756994 (+ 43us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 06:07:20.757088 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:07:20.757092 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:07:20.757807 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:07:20.757810 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:07:20.758827 (+ 1017us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:07:20.758852 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:07:20.759537 (+ 685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:07:20.759542 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:07:20.759652 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:07:20.759668 (+ 16us) client_negotiation.cc:769] Sending connection context
0620 06:07:20.759765 (+ 97us) client_negotiation.cc:240] Negotiation successful
0620 06:07:20.759838 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":316,"mutex_wait_us":90,"spinlock_wait_cycles":3072,"thread_start_us":82,"threads_started":1}
I20260620 06:07:20.760540 19341 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "09d9ebc901d1409db79307028aa69ffa" candidate_uuid: "4fe4e45f6e02471594ff656d8aff63ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fe86b816edb423498c5dd937c81279c" is_pre_election: true
I20260620 06:07:20.760672 19341 raft_consensus.cc:2454] T 09d9ebc901d1409db79307028aa69ffa P 7fe86b816edb423498c5dd937c81279c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4fe4e45f6e02471594ff656d8aff63ac in term 0.
I20260620 06:07:20.760991 19621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "09d9ebc901d1409db79307028aa69ffa" candidate_uuid: "4fe4e45f6e02471594ff656d8aff63ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2316ce59fd744c01a82cb798c635b913" is_pre_election: true
I20260620 06:07:20.761052 19413 leader_election.cc:302] T 09d9ebc901d1409db79307028aa69ffa P 4fe4e45f6e02471594ff656d8aff63ac [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4fe4e45f6e02471594ff656d8aff63ac, 7fe86b816edb423498c5dd937c81279c; no voters:
I20260620 06:07:20.761221 19621 raft_consensus.cc:2454] T 09d9ebc901d1409db79307028aa69ffa P 2316ce59fd744c01a82cb798c635b913 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4fe4e45f6e02471594ff656d8aff63ac in term 0.
I20260620 06:07:20.761286 19702 raft_consensus.cc:2781] T 09d9ebc901d1409db79307028aa69ffa P 4fe4e45f6e02471594ff656d8aff63ac [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 06:07:20.761374 19702 raft_consensus.cc:482] T 09d9ebc901d1409db79307028aa69ffa P 4fe4e45f6e02471594ff656d8aff63ac [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 06:07:20.761421 19702 raft_consensus.cc:3037] T 09d9ebc901d1409db79307028aa69ffa P 4fe4e45f6e02471594ff656d8aff63ac [term 0 FOLLOWER]: Advancing to term 1
I20260620 06:07:20.762383 19702 raft_consensus.cc:504] T 09d9ebc901d1409db79307028aa69ffa P 4fe4e45f6e02471594ff656d8aff63ac [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fe86b816edb423498c5dd937c81279c" member_type: VOTER last_known_addr { host: "127.11.102.1" port: 33553 } } peers { permanent_uuid: "4fe4e45f6e02471594ff656d8aff63ac" member_type: VOTER last_known_addr { host: "127.11.102.2" port: 38643 } } peers { permanent_uuid: "2316ce59fd744c01a82cb798c635b913" member_type: VOTER last_known_addr { host: "127.11.102.3" port: 37845 } }
I20260620 06:07:20.762764 19702 leader_election.cc:288] T 09d9ebc901d1409db79307028aa69ffa P 4fe4e45f6e02471594ff656d8aff63ac [CANDIDATE]: Term 1 election: Requested vote from peers 7fe86b816edb423498c5dd937c81279c (127.11.102.1:33553), 2316ce59fd744c01a82cb798c635b913 (127.11.102.3:37845)
I20260620 06:07:20.763083 19341 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "09d9ebc901d1409db79307028aa69ffa" candidate_uuid: "4fe4e45f6e02471594ff656d8aff63ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fe86b816edb423498c5dd937c81279c"
I20260620 06:07:20.763145 19621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "09d9ebc901d1409db79307028aa69ffa" candidate_uuid: "4fe4e45f6e02471594ff656d8aff63ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2316ce59fd744c01a82cb798c635b913"
I20260620 06:07:20.763190 19341 raft_consensus.cc:3037] T 09d9ebc901d1409db79307028aa69ffa P 7fe86b816edb423498c5dd937c81279c [term 0 FOLLOWER]: Advancing to term 1
I20260620 06:07:20.763281 19621 raft_consensus.cc:3037] T 09d9ebc901d1409db79307028aa69ffa P 2316ce59fd744c01a82cb798c635b913 [term 0 FOLLOWER]: Advancing to term 1
I20260620 06:07:20.764145 19341 raft_consensus.cc:2454] T 09d9ebc901d1409db79307028aa69ffa P 7fe86b816edb423498c5dd937c81279c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4fe4e45f6e02471594ff656d8aff63ac in term 1.
I20260620 06:07:20.764300 19621 raft_consensus.cc:2454] T 09d9ebc901d1409db79307028aa69ffa P 2316ce59fd744c01a82cb798c635b913 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4fe4e45f6e02471594ff656d8aff63ac in term 1.
I20260620 06:07:20.764423 19413 leader_election.cc:302] T 09d9ebc901d1409db79307028aa69ffa P 4fe4e45f6e02471594ff656d8aff63ac [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4fe4e45f6e02471594ff656d8aff63ac, 7fe86b816edb423498c5dd937c81279c; no voters:
I20260620 06:07:20.764619 19702 raft_consensus.cc:2781] T 09d9ebc901d1409db79307028aa69ffa P 4fe4e45f6e02471594ff656d8aff63ac [term 1 FOLLOWER]: Leader election won for term 1
I20260620 06:07:20.764775 19702 raft_consensus.cc:686] T 09d9ebc901d1409db79307028aa69ffa P 4fe4e45f6e02471594ff656d8aff63ac [term 1 LEADER]: Becoming Leader. State: Replica: 4fe4e45f6e02471594ff656d8aff63ac, State: Running, Role: LEADER
I20260620 06:07:20.765018 19702 consensus_queue.cc:237] T 09d9ebc901d1409db79307028aa69ffa P 4fe4e45f6e02471594ff656d8aff63ac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fe86b816edb423498c5dd937c81279c" member_type: VOTER last_known_addr { host: "127.11.102.1" port: 33553 } } peers { permanent_uuid: "4fe4e45f6e02471594ff656d8aff63ac" member_type: VOTER last_known_addr { host: "127.11.102.2" port: 38643 } } peers { permanent_uuid: "2316ce59fd744c01a82cb798c635b913" member_type: VOTER last_known_addr { host: "127.11.102.3" port: 37845 } }
I20260620 06:07:20.767055 19184 catalog_manager.cc:5495] T 09d9ebc901d1409db79307028aa69ffa P 4fe4e45f6e02471594ff656d8aff63ac reported cstate change: term changed from 0 to 1, leader changed from <none> to 4fe4e45f6e02471594ff656d8aff63ac (127.11.102.2). New cstate: current_term: 1 leader_uuid: "4fe4e45f6e02471594ff656d8aff63ac" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fe86b816edb423498c5dd937c81279c" member_type: VOTER last_known_addr { host: "127.11.102.1" port: 33553 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4fe4e45f6e02471594ff656d8aff63ac" member_type: VOTER last_known_addr { host: "127.11.102.2" port: 38643 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2316ce59fd744c01a82cb798c635b913" member_type: VOTER last_known_addr { host: "127.11.102.3" port: 37845 } health_report { overall_health: UNKNOWN } } }
Jun 20 06:07:20 dist-test-slave-9hm5 krb5kdc[19139](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935639, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.102.62@KRBTEST.COM for oryx/127.11.102.62@KRBTEST.COM
I20260620 06:07:20.813458 19732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:07:20.805159 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.102.62:36245 (local address 127.0.0.1:56278)
0620 06:07:20.805444 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0620 06:07:20.805456 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 06:07:20.805550 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:07:20.805775 (+ 225us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:07:20.805777 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:07:20.806328 (+ 551us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:07:20.806418 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:07:20.806424 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:07:20.807178 (+ 754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:07:20.807180 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:07:20.807817 (+ 637us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:07:20.807826 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:07:20.808234 (+ 408us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:07:20.808237 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:07:20.808313 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:07:20.808811 (+ 498us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:07:20.808823 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:07:20.810411 (+ 1588us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:07:20.812061 (+ 1650us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:07:20.812070 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:07:20.812075 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:07:20.812329 (+ 254us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:07:20.812565 (+ 236us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:07:20.812567 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:07:20.812568 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:07:20.812651 (+ 83us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:07:20.813002 (+ 351us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:07:20.813008 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:07:20.813106 (+ 98us) client_negotiation.cc:769] Sending connection context
0620 06:07:20.813205 (+ 99us) client_negotiation.cc:240] Negotiation successful
0620 06:07:20.813327 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":207,"thread_start_us":124,"threads_started":1}
I20260620 06:07:20.813519 19233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:07:20.805238 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56278 (local address 127.11.102.62:36245)
0620 06:07:20.805378 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0620 06:07:20.805382 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 06:07:20.805531 (+ 149us) server_negotiation.cc:407] Connection header received
0620 06:07:20.805631 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:07:20.805633 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:07:20.805671 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:07:20.805738 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:07:20.806543 (+ 805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:07:20.807063 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:07:20.807933 (+ 870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:07:20.808138 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:07:20.808220 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:07:20.810561 (+ 2341us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:07:20.810564 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:07:20.810566 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:07:20.810587 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:07:20.811964 (+ 1377us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:07:20.812418 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:07:20.812420 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:07:20.812421 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:07:20.812487 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:07:20.812729 (+ 242us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:07:20.812731 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:07:20.812731 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:07:20.812891 (+ 160us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:07:20.812964 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0620 06:07:20.813253 (+ 289us) server_negotiation.cc:299] Negotiation successful
0620 06:07:20.813394 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":50,"server-negotiator.queue_time_us":69}
I20260620 06:07:20.817209 19184 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.11.102.62@KRBTEST.COM'} at 127.0.0.1:56278:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260620 06:07:20.817987 19184 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 06:07:20.822619 19461 tablet_service.cc:1467] Processing CreateTablet for tablet 6995327a20334d50bfcdc8b70ffb4a28 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=ba89173926c94497aefa69d9e950846a]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 06:07:20.822611 19321 tablet_service.cc:1467] Processing CreateTablet for tablet 6995327a20334d50bfcdc8b70ffb4a28 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=ba89173926c94497aefa69d9e950846a]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 06:07:20.822796 19601 tablet_service.cc:1467] Processing CreateTablet for tablet 6995327a20334d50bfcdc8b70ffb4a28 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=ba89173926c94497aefa69d9e950846a]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 06:07:20.823009 19461 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6995327a20334d50bfcdc8b70ffb4a28. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 06:07:20.823025 19321 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6995327a20334d50bfcdc8b70ffb4a28. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 06:07:20.823099 19601 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6995327a20334d50bfcdc8b70ffb4a28. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 06:07:20.825472 19696 tablet_bootstrap.cc:492] T 6995327a20334d50bfcdc8b70ffb4a28 P 7fe86b816edb423498c5dd937c81279c: Bootstrap starting.
I20260620 06:07:20.825623 19697 tablet_bootstrap.cc:492] T 6995327a20334d50bfcdc8b70ffb4a28 P 2316ce59fd744c01a82cb798c635b913: Bootstrap starting.
I20260620 06:07:20.825955 19695 tablet_bootstrap.cc:492] T 6995327a20334d50bfcdc8b70ffb4a28 P 4fe4e45f6e02471594ff656d8aff63ac: Bootstrap starting.
I20260620 06:07:20.826462 19697 tablet_bootstrap.cc:654] T 6995327a20334d50bfcdc8b70ffb4a28 P 2316ce59fd744c01a82cb798c635b913: Neither blocks nor log segments found. Creating new log.
I20260620 06:07:20.826495 19696 tablet_bootstrap.cc:654] T 6995327a20334d50bfcdc8b70ffb4a28 P 7fe86b816edb423498c5dd937c81279c: Neither blocks nor log segments found. Creating new log.
I20260620 06:07:20.826884 19695 tablet_bootstrap.cc:654] T 6995327a20334d50bfcdc8b70ffb4a28 P 4fe4e45f6e02471594ff656d8aff63ac: Neither blocks nor log segments found. Creating new log.
I20260620 06:07:20.827618 19697 tablet_bootstrap.cc:492] T 6995327a20334d50bfcdc8b70ffb4a28 P 2316ce59fd744c01a82cb798c635b913: No bootstrap required, opened a new log
I20260620 06:07:20.827749 19697 ts_tablet_manager.cc:1397] T 6995327a20334d50bfcdc8b70ffb4a28 P 2316ce59fd744c01a82cb798c635b913: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260620 06:07:20.827989 19696 tablet_bootstrap.cc:492] T 6995327a20334d50bfcdc8b70ffb4a28 P 7fe86b816edb423498c5dd937c81279c: No bootstrap required, opened a new log
I20260620 06:07:20.828073 19696 ts_tablet_manager.cc:1397] T 6995327a20334d50bfcdc8b70ffb4a28 P 7fe86b816edb423498c5dd937c81279c: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20260620 06:07:20.828207 19697 raft_consensus.cc:348] T 6995327a20334d50bfcdc8b70ffb4a28 P 2316ce59fd744c01a82cb798c635b913 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fe86b816edb423498c5dd937c81279c" member_type: VOTER last_known_addr { host: "127.11.102.1" port: 33553 } } peers { permanent_uuid: "4fe4e45f6e02471594ff656d8aff63ac" member_type: VOTER last_known_addr { host: "127.11.102.2" port: 38643 } } peers { permanent_uuid: "2316ce59fd744c01a82cb798c635b913" member_type: VOTER last_known_addr { host: "127.11.102.3" port: 37845 } }
I20260620 06:07:20.828323 19697 raft_consensus.cc:374] T 6995327a20334d50bfcdc8b70ffb4a28 P 2316ce59fd744c01a82cb798c635b913 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 06:07:20.828326 19695 tablet_bootstrap.cc:492] T 6995327a20334d50bfcdc8b70ffb4a28 P 4fe4e45f6e02471594ff656d8aff63ac: No bootstrap required, opened a new log
I20260620 06:07:20.828362 19697 raft_consensus.cc:729] T 6995327a20334d50bfcdc8b70ffb4a28 P 2316ce59fd744c01a82cb798c635b913 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2316ce59fd744c01a82cb798c635b913, State: Initialized, Role: FOLLOWER
I20260620 06:07:20.828395 19695 ts_tablet_manager.cc:1397] T 6995327a20334d50bfcdc8b70ffb4a28 P 4fe4e45f6e02471594ff656d8aff63ac: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260620 06:07:20.828493 19697 consensus_queue.cc:260] T 6995327a20334d50bfcdc8b70ffb4a28 P 2316ce59fd744c01a82cb798c635b913 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fe86b816edb423498c5dd937c81279c" member_type: VOTER last_known_addr { host: "127.11.102.1" port: 33553 } } peers { permanent_uuid: "4fe4e45f6e02471594ff656d8aff63ac" member_type: VOTER last_known_addr { host: "127.11.102.2" port: 38643 } } peers { permanent_uuid: "2316ce59fd744c01a82cb798c635b913" member_type: VOTER last_known_addr { host: "127.11.102.3" port: 37845 } }
I20260620 06:07:20.828534 19696 raft_consensus.cc:348] T 6995327a20334d50bfcdc8b70ffb4a28 P 7fe86b816edb423498c5dd937c81279c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fe86b816edb423498c5dd937c81279c" member_type: VOTER last_known_addr { host: "127.11.102.1" port: 33553 } } peers { permanent_uuid: "4fe4e45f6e02471594ff656d8aff63ac" member_type: VOTER last_known_addr { host: "127.11.102.2" port: 38643 } } peers { permanent_uuid: "2316ce59fd744c01a82cb798c635b913" member_type: VOTER last_known_addr { host: "127.11.102.3" port: 37845 } }
I20260620 06:07:20.828634 19696 raft_consensus.cc:374] T 6995327a20334d50bfcdc8b70ffb4a28 P 7fe86b816edb423498c5dd937c81279c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 06:07:20.828676 19696 raft_consensus.cc:729] T 6995327a20334d50bfcdc8b70ffb4a28 P 7fe86b816edb423498c5dd937c81279c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7fe86b816edb423498c5dd937c81279c, State: Initialized, Role: FOLLOWER
I20260620 06:07:20.828787 19697 ts_tablet_manager.cc:1428] T 6995327a20334d50bfcdc8b70ffb4a28 P 2316ce59fd744c01a82cb798c635b913: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260620 06:07:20.828794 19696 consensus_queue.cc:260] T 6995327a20334d50bfcdc8b70ffb4a28 P 7fe86b816edb423498c5dd937c81279c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fe86b816edb423498c5dd937c81279c" member_type: VOTER last_known_addr { host: "127.11.102.1" port: 33553 } } peers { permanent_uuid: "4fe4e45f6e02471594ff656d8aff63ac" member_type: VOTER last_known_addr { host: "127.11.102.2" port: 38643 } } peers { permanent_uuid: "2316ce59fd744c01a82cb798c635b913" member_type: VOTER last_known_addr { host: "127.11.102.3" port: 37845 } }
I20260620 06:07:20.828821 19695 raft_consensus.cc:348] T 6995327a20334d50bfcdc8b70ffb4a28 P 4fe4e45f6e02471594ff656d8aff63ac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fe86b816edb423498c5dd937c81279c" member_type: VOTER last_known_addr { host: "127.11.102.1" port: 33553 } } peers { permanent_uuid: "4fe4e45f6e02471594ff656d8aff63ac" member_type: VOTER last_known_addr { host: "127.11.102.2" port: 38643 } } peers { permanent_uuid: "2316ce59fd744c01a82cb798c635b913" member_type: VOTER last_known_addr { host: "127.11.102.3" port: 37845 } }
I20260620 06:07:20.828910 19695 raft_consensus.cc:374] T 6995327a20334d50bfcdc8b70ffb4a28 P 4fe4e45f6e02471594ff656d8aff63ac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 06:07:20.828939 19695 raft_consensus.cc:729] T 6995327a20334d50bfcdc8b70ffb4a28 P 4fe4e45f6e02471594ff656d8aff63ac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4fe4e45f6e02471594ff656d8aff63ac, State: Initialized, Role: FOLLOWER
I20260620 06:07:20.829051 19696 ts_tablet_manager.cc:1428] T 6995327a20334d50bfcdc8b70ffb4a28 P 7fe86b816edb423498c5dd937c81279c: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260620 06:07:20.829003 19701 tablet_replica.cc:439] T 6995327a20334d50bfcdc8b70ffb4a28 P 2316ce59fd744c01a82cb798c635b913: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fe86b816edb423498c5dd937c81279c" member_type: VOTER last_known_addr { host: "127.11.102.1" port: 33553 } } peers { permanent_uuid: "4fe4e45f6e02471594ff656d8aff63ac" member_type: VOTER last_known_addr { host: "127.11.102.2" port: 38643 } } peers { permanent_uuid: "2316ce59fd744c01a82cb798c635b913" member_type: VOTER last_known_addr { host: "127.11.102.3" port: 37845 } } }
I20260620 06:07:20.829043 19695 consensus_queue.cc:260] T 6995327a20334d50bfcdc8b70ffb4a28 P 4fe4e45f6e02471594ff656d8aff63ac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fe86b816edb423498c5dd937c81279c" member_type: VOTER last_known_addr { host: "127.11.102.1" port: 33553 } } peers { permanent_uuid: "4fe4e45f6e02471594ff656d8aff63ac" member_type: VOTER last_known_addr { host: "127.11.102.2" port: 38643 } } peers { permanent_uuid: "2316ce59fd744c01a82cb798c635b913" member_type: VOTER last_known_addr { host: "127.11.102.3" port: 37845 } }
I20260620 06:07:20.829110 19701 tablet_replica.cc:442] T 6995327a20334d50bfcdc8b70ffb4a28 P 2316ce59fd744c01a82cb798c635b913: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 06:07:20.829279 19695 ts_tablet_manager.cc:1428] T 6995327a20334d50bfcdc8b70ffb4a28 P 4fe4e45f6e02471594ff656d8aff63ac: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260620 06:07:20.829492 19703 tablet_replica.cc:439] T 6995327a20334d50bfcdc8b70ffb4a28 P 7fe86b816edb423498c5dd937c81279c: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fe86b816edb423498c5dd937c81279c" member_type: VOTER last_known_addr { host: "127.11.102.1" port: 33553 } } peers { permanent_uuid: "4fe4e45f6e02471594ff656d8aff63ac" member_type: VOTER last_known_addr { host: "127.11.102.2" port: 38643 } } peers { permanent_uuid: "2316ce59fd744c01a82cb798c635b913" member_type: VOTER last_known_addr { host: "127.11.102.3" port: 37845 } } }
I20260620 06:07:20.829615 19703 tablet_replica.cc:442] T 6995327a20334d50bfcdc8b70ffb4a28 P 7fe86b816edb423498c5dd937c81279c: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 06:07:20.829818 19702 tablet_replica.cc:439] T 6995327a20334d50bfcdc8b70ffb4a28 P 4fe4e45f6e02471594ff656d8aff63ac: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fe86b816edb423498c5dd937c81279c" member_type: VOTER last_known_addr { host: "127.11.102.1" port: 33553 } } peers { permanent_uuid: "4fe4e45f6e02471594ff656d8aff63ac" member_type: VOTER last_known_addr { host: "127.11.102.2" port: 38643 } } peers { permanent_uuid: "2316ce59fd744c01a82cb798c635b913" member_type: VOTER last_known_addr { host: "127.11.102.3" port: 37845 } } }
I20260620 06:07:20.829909 19702 tablet_replica.cc:442] T 6995327a20334d50bfcdc8b70ffb4a28 P 4fe4e45f6e02471594ff656d8aff63ac: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 06:07:20.963488 19703 raft_consensus.cc:482] T 6995327a20334d50bfcdc8b70ffb4a28 P 7fe86b816edb423498c5dd937c81279c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 06:07:20.963693 19703 raft_consensus.cc:504] T 6995327a20334d50bfcdc8b70ffb4a28 P 7fe86b816edb423498c5dd937c81279c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fe86b816edb423498c5dd937c81279c" member_type: VOTER last_known_addr { host: "127.11.102.1" port: 33553 } } peers { permanent_uuid: "4fe4e45f6e02471594ff656d8aff63ac" member_type: VOTER last_known_addr { host: "127.11.102.2" port: 38643 } } peers { permanent_uuid: "2316ce59fd744c01a82cb798c635b913" member_type: VOTER last_known_addr { host: "127.11.102.3" port: 37845 } }
I20260620 06:07:20.964823 19703 leader_election.cc:288] T 6995327a20334d50bfcdc8b70ffb4a28 P 7fe86b816edb423498c5dd937c81279c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4fe4e45f6e02471594ff656d8aff63ac (127.11.102.2:38643), 2316ce59fd744c01a82cb798c635b913 (127.11.102.3:37845)
I20260620 06:07:20.968811 19736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:07:20.965098 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.102.3:37845 (local address 127.11.102.1:46883)
0620 06:07:20.965457 (+ 359us) negotiation.cc:107] Waiting for socket to connect
0620 06:07:20.965474 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 06:07:20.965620 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:07:20.965874 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:07:20.965877 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:07:20.965891 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 06:07:20.965992 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:07:20.966004 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:07:20.966704 (+ 700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:07:20.966710 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:07:20.967702 (+ 992us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:07:20.967716 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:07:20.968349 (+ 633us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:07:20.968353 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:07:20.968447 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:07:20.968475 (+ 28us) client_negotiation.cc:769] Sending connection context
0620 06:07:20.968553 (+ 78us) client_negotiation.cc:240] Negotiation successful
0620 06:07:20.968619 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":295,"mutex_wait_us":50,"thread_start_us":122,"threads_started":1}
I20260620 06:07:20.968842 19693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:07:20.965255 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.102.1:46883 (local address 127.11.102.3:37845)
0620 06:07:20.965435 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0620 06:07:20.965441 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 06:07:20.965562 (+ 121us) server_negotiation.cc:407] Connection header received
0620 06:07:20.965730 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:07:20.965733 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:07:20.965783 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:07:20.965864 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 06:07:20.966100 (+ 236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:07:20.966598 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:07:20.967821 (+ 1223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:07:20.968229 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:07:20.968359 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:07:20.968434 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0620 06:07:20.968594 (+ 160us) server_negotiation.cc:299] Negotiation successful
0620 06:07:20.968707 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260620 06:07:20.968947 19738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:07:20.965210 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.102.1:36149 (local address 127.11.102.2:38643)
0620 06:07:20.965506 (+ 296us) server_negotiation.cc:206] Beginning negotiation
0620 06:07:20.965512 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 06:07:20.965651 (+ 139us) server_negotiation.cc:407] Connection header received
0620 06:07:20.965822 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:07:20.965826 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:07:20.965879 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:07:20.965998 (+ 119us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 06:07:20.966252 (+ 254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:07:20.966763 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:07:20.967986 (+ 1223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:07:20.968414 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:07:20.968564 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:07:20.968621 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0620 06:07:20.968731 (+ 110us) server_negotiation.cc:299] Negotiation successful
0620 06:07:20.968803 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":210,"thread_start_us":68,"threads_started":1}
I20260620 06:07:20.969066 19737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:07:20.965123 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.102.2:38643 (local address 127.11.102.1:36149)
0620 06:07:20.965521 (+ 398us) negotiation.cc:107] Waiting for socket to connect
0620 06:07:20.965537 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 06:07:20.965707 (+ 170us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:07:20.966007 (+ 300us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:07:20.966010 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:07:20.966026 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 06:07:20.966126 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:07:20.966133 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:07:20.966919 (+ 786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:07:20.966922 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:07:20.967855 (+ 933us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:07:20.967864 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:07:20.968533 (+ 669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:07:20.968539 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:07:20.968619 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:07:20.968637 (+ 18us) client_negotiation.cc:769] Sending connection context
0620 06:07:20.968705 (+ 68us) client_negotiation.cc:240] Negotiation successful
0620 06:07:20.968755 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":334,"thread_start_us":108,"threads_started":1}
I20260620 06:07:20.969483 19471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6995327a20334d50bfcdc8b70ffb4a28" candidate_uuid: "7fe86b816edb423498c5dd937c81279c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4fe4e45f6e02471594ff656d8aff63ac" is_pre_election: true
I20260620 06:07:20.969497 19621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6995327a20334d50bfcdc8b70ffb4a28" candidate_uuid: "7fe86b816edb423498c5dd937c81279c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2316ce59fd744c01a82cb798c635b913" is_pre_election: true
I20260620 06:07:20.969610 19471 raft_consensus.cc:2454] T 6995327a20334d50bfcdc8b70ffb4a28 P 4fe4e45f6e02471594ff656d8aff63ac [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7fe86b816edb423498c5dd937c81279c in term 0.
I20260620 06:07:20.969663 19621 raft_consensus.cc:2454] T 6995327a20334d50bfcdc8b70ffb4a28 P 2316ce59fd744c01a82cb798c635b913 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7fe86b816edb423498c5dd937c81279c in term 0.
I20260620 06:07:20.969974 19275 leader_election.cc:302] T 6995327a20334d50bfcdc8b70ffb4a28 P 7fe86b816edb423498c5dd937c81279c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 2316ce59fd744c01a82cb798c635b913, 4fe4e45f6e02471594ff656d8aff63ac, 7fe86b816edb423498c5dd937c81279c; no voters:
I20260620 06:07:20.970293 19703 raft_consensus.cc:2781] T 6995327a20334d50bfcdc8b70ffb4a28 P 7fe86b816edb423498c5dd937c81279c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 06:07:20.970374 19703 raft_consensus.cc:482] T 6995327a20334d50bfcdc8b70ffb4a28 P 7fe86b816edb423498c5dd937c81279c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 06:07:20.970434 19703 raft_consensus.cc:3037] T 6995327a20334d50bfcdc8b70ffb4a28 P 7fe86b816edb423498c5dd937c81279c [term 0 FOLLOWER]: Advancing to term 1
I20260620 06:07:20.971413 19703 raft_consensus.cc:504] T 6995327a20334d50bfcdc8b70ffb4a28 P 7fe86b816edb423498c5dd937c81279c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fe86b816edb423498c5dd937c81279c" member_type: VOTER last_known_addr { host: "127.11.102.1" port: 33553 } } peers { permanent_uuid: "4fe4e45f6e02471594ff656d8aff63ac" member_type: VOTER last_known_addr { host: "127.11.102.2" port: 38643 } } peers { permanent_uuid: "2316ce59fd744c01a82cb798c635b913" member_type: VOTER last_known_addr { host: "127.11.102.3" port: 37845 } }
I20260620 06:07:20.971854 19703 leader_election.cc:288] T 6995327a20334d50bfcdc8b70ffb4a28 P 7fe86b816edb423498c5dd937c81279c [CANDIDATE]: Term 1 election: Requested vote from peers 4fe4e45f6e02471594ff656d8aff63ac (127.11.102.2:38643), 2316ce59fd744c01a82cb798c635b913 (127.11.102.3:37845)
I20260620 06:07:20.972180 19471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6995327a20334d50bfcdc8b70ffb4a28" candidate_uuid: "7fe86b816edb423498c5dd937c81279c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4fe4e45f6e02471594ff656d8aff63ac"
I20260620 06:07:20.972316 19471 raft_consensus.cc:3037] T 6995327a20334d50bfcdc8b70ffb4a28 P 4fe4e45f6e02471594ff656d8aff63ac [term 0 FOLLOWER]: Advancing to term 1
I20260620 06:07:20.973407 19471 raft_consensus.cc:2454] T 6995327a20334d50bfcdc8b70ffb4a28 P 4fe4e45f6e02471594ff656d8aff63ac [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7fe86b816edb423498c5dd937c81279c in term 1.
I20260620 06:07:20.973660 19621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6995327a20334d50bfcdc8b70ffb4a28" candidate_uuid: "7fe86b816edb423498c5dd937c81279c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2316ce59fd744c01a82cb798c635b913"
I20260620 06:07:20.973691 19275 leader_election.cc:302] T 6995327a20334d50bfcdc8b70ffb4a28 P 7fe86b816edb423498c5dd937c81279c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4fe4e45f6e02471594ff656d8aff63ac, 7fe86b816edb423498c5dd937c81279c; no voters:
I20260620 06:07:20.973793 19621 raft_consensus.cc:3037] T 6995327a20334d50bfcdc8b70ffb4a28 P 2316ce59fd744c01a82cb798c635b913 [term 0 FOLLOWER]: Advancing to term 1
I20260620 06:07:20.973891 19703 raft_consensus.cc:2781] T 6995327a20334d50bfcdc8b70ffb4a28 P 7fe86b816edb423498c5dd937c81279c [term 1 FOLLOWER]: Leader election won for term 1
I20260620 06:07:20.974080 19703 raft_consensus.cc:686] T 6995327a20334d50bfcdc8b70ffb4a28 P 7fe86b816edb423498c5dd937c81279c [term 1 LEADER]: Becoming Leader. State: Replica: 7fe86b816edb423498c5dd937c81279c, State: Running, Role: LEADER
I20260620 06:07:20.974345 19703 consensus_queue.cc:237] T 6995327a20334d50bfcdc8b70ffb4a28 P 7fe86b816edb423498c5dd937c81279c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fe86b816edb423498c5dd937c81279c" member_type: VOTER last_known_addr { host: "127.11.102.1" port: 33553 } } peers { permanent_uuid: "4fe4e45f6e02471594ff656d8aff63ac" member_type: VOTER last_known_addr { host: "127.11.102.2" port: 38643 } } peers { permanent_uuid: "2316ce59fd744c01a82cb798c635b913" member_type: VOTER last_known_addr { host: "127.11.102.3" port: 37845 } }
I20260620 06:07:20.974988 19621 raft_consensus.cc:2454] T 6995327a20334d50bfcdc8b70ffb4a28 P 2316ce59fd744c01a82cb798c635b913 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7fe86b816edb423498c5dd937c81279c in term 1.
I20260620 06:07:20.975718 19739 tablet_replica.cc:439] T 6995327a20334d50bfcdc8b70ffb4a28 P 7fe86b816edb423498c5dd937c81279c: TxnStatusTablet state changed. Reason: New leader 7fe86b816edb423498c5dd937c81279c. Latest consensus state: current_term: 1 leader_uuid: "7fe86b816edb423498c5dd937c81279c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fe86b816edb423498c5dd937c81279c" member_type: VOTER last_known_addr { host: "127.11.102.1" port: 33553 } } peers { permanent_uuid: "4fe4e45f6e02471594ff656d8aff63ac" member_type: VOTER last_known_addr { host: "127.11.102.2" port: 38643 } } peers { permanent_uuid: "2316ce59fd744c01a82cb798c635b913" member_type: VOTER last_known_addr { host: "127.11.102.3" port: 37845 } } }
I20260620 06:07:20.975838 19739 tablet_replica.cc:442] T 6995327a20334d50bfcdc8b70ffb4a28 P 7fe86b816edb423498c5dd937c81279c: This TxnStatusTablet replica's current role is: LEADER
I20260620 06:07:20.976120 19741 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260620 06:07:20.976269 19741 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260620 06:07:20.976948 19184 catalog_manager.cc:5495] T 6995327a20334d50bfcdc8b70ffb4a28 P 7fe86b816edb423498c5dd937c81279c reported cstate change: term changed from 0 to 1, leader changed from <none> to 7fe86b816edb423498c5dd937c81279c (127.11.102.1). New cstate: current_term: 1 leader_uuid: "7fe86b816edb423498c5dd937c81279c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fe86b816edb423498c5dd937c81279c" member_type: VOTER last_known_addr { host: "127.11.102.1" port: 33553 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4fe4e45f6e02471594ff656d8aff63ac" member_type: VOTER last_known_addr { host: "127.11.102.2" port: 38643 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2316ce59fd744c01a82cb798c635b913" member_type: VOTER last_known_addr { host: "127.11.102.3" port: 37845 } health_report { overall_health: UNKNOWN } } }
I20260620 06:07:21.021700 19692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:07:21.017950 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52906 (local address 127.11.102.1:33553)
0620 06:07:21.018083 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0620 06:07:21.018087 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 06:07:21.018102 (+ 15us) server_negotiation.cc:407] Connection header received
0620 06:07:21.018276 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:07:21.018285 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:07:21.018340 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:07:21.018455 (+ 115us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 06:07:21.018687 (+ 232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:07:21.019208 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:07:21.020135 (+ 927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:07:21.020354 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:07:21.020466 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:07:21.020821 (+ 355us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 06:07:21.021268 (+ 447us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 06:07:21.021382 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0620 06:07:21.021502 (+ 120us) server_negotiation.cc:299] Negotiation successful
0620 06:07:21.021549 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260620 06:07:21.021741 19732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:07:21.017839 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.102.1:33553 (local address 127.0.0.1:52906)
0620 06:07:21.017969 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0620 06:07:21.017986 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 06:07:21.018102 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:07:21.018466 (+ 364us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:07:21.018469 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:07:21.018475 (+ 6us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 06:07:21.018580 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:07:21.018585 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:07:21.019317 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:07:21.019323 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:07:21.020011 (+ 688us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:07:21.020020 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:07:21.020463 (+ 443us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:07:21.020467 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:07:21.020574 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:07:21.020662 (+ 88us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 06:07:21.021379 (+ 717us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 06:07:21.021395 (+ 16us) client_negotiation.cc:769] Sending connection context
0620 06:07:21.021483 (+ 88us) client_negotiation.cc:240] Negotiation successful
0620 06:07:21.021569 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30}
I20260620 06:07:21.024614 19471 raft_consensus.cc:1261] T 6995327a20334d50bfcdc8b70ffb4a28 P 4fe4e45f6e02471594ff656d8aff63ac [term 1 FOLLOWER]: Refusing update from remote peer 7fe86b816edb423498c5dd937c81279c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 06:07:21.024646 19621 raft_consensus.cc:1261] T 6995327a20334d50bfcdc8b70ffb4a28 P 2316ce59fd744c01a82cb798c635b913 [term 1 FOLLOWER]: Refusing update from remote peer 7fe86b816edb423498c5dd937c81279c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 06:07:21.025131 19703 consensus_queue.cc:1035] T 6995327a20334d50bfcdc8b70ffb4a28 P 7fe86b816edb423498c5dd937c81279c [LEADER]: Connected to new peer: Peer: permanent_uuid: "4fe4e45f6e02471594ff656d8aff63ac" member_type: VOTER last_known_addr { host: "127.11.102.2" port: 38643 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 06:07:21.025389 19739 consensus_queue.cc:1035] T 6995327a20334d50bfcdc8b70ffb4a28 P 7fe86b816edb423498c5dd937c81279c [LEADER]: Connected to new peer: Peer: permanent_uuid: "2316ce59fd744c01a82cb798c635b913" member_type: VOTER last_known_addr { host: "127.11.102.3" port: 37845 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 06:07:21.026911 19701 tablet_replica.cc:439] T 6995327a20334d50bfcdc8b70ffb4a28 P 2316ce59fd744c01a82cb798c635b913: TxnStatusTablet state changed. Reason: New leader 7fe86b816edb423498c5dd937c81279c. Latest consensus state: current_term: 1 leader_uuid: "7fe86b816edb423498c5dd937c81279c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fe86b816edb423498c5dd937c81279c" member_type: VOTER last_known_addr { host: "127.11.102.1" port: 33553 } } peers { permanent_uuid: "4fe4e45f6e02471594ff656d8aff63ac" member_type: VOTER last_known_addr { host: "127.11.102.2" port: 38643 } } peers { permanent_uuid: "2316ce59fd744c01a82cb798c635b913" member_type: VOTER last_known_addr { host: "127.11.102.3" port: 37845 } } }
I20260620 06:07:21.027036 19701 tablet_replica.cc:442] T 6995327a20334d50bfcdc8b70ffb4a28 P 2316ce59fd744c01a82cb798c635b913: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 06:07:21.027500 19702 tablet_replica.cc:439] T 6995327a20334d50bfcdc8b70ffb4a28 P 4fe4e45f6e02471594ff656d8aff63ac: TxnStatusTablet state changed. Reason: New leader 7fe86b816edb423498c5dd937c81279c. Latest consensus state: current_term: 1 leader_uuid: "7fe86b816edb423498c5dd937c81279c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fe86b816edb423498c5dd937c81279c" member_type: VOTER last_known_addr { host: "127.11.102.1" port: 33553 } } peers { permanent_uuid: "4fe4e45f6e02471594ff656d8aff63ac" member_type: VOTER last_known_addr { host: "127.11.102.2" port: 38643 } } peers { permanent_uuid: "2316ce59fd744c01a82cb798c635b913" member_type: VOTER last_known_addr { host: "127.11.102.3" port: 37845 } } }
I20260620 06:07:21.028108 19702 tablet_replica.cc:442] T 6995327a20334d50bfcdc8b70ffb4a28 P 4fe4e45f6e02471594ff656d8aff63ac: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 06:07:21.029189 19739 tablet_replica.cc:439] T 6995327a20334d50bfcdc8b70ffb4a28 P 7fe86b816edb423498c5dd937c81279c: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "7fe86b816edb423498c5dd937c81279c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fe86b816edb423498c5dd937c81279c" member_type: VOTER last_known_addr { host: "127.11.102.1" port: 33553 } } peers { permanent_uuid: "4fe4e45f6e02471594ff656d8aff63ac" member_type: VOTER last_known_addr { host: "127.11.102.2" port: 38643 } } peers { permanent_uuid: "2316ce59fd744c01a82cb798c635b913" member_type: VOTER last_known_addr { host: "127.11.102.3" port: 37845 } } }
I20260620 06:07:21.029299 19739 tablet_replica.cc:442] T 6995327a20334d50bfcdc8b70ffb4a28 P 7fe86b816edb423498c5dd937c81279c: This TxnStatusTablet replica's current role is: LEADER
I20260620 06:07:21.029976 19739 tablet_replica.cc:439] T 6995327a20334d50bfcdc8b70ffb4a28 P 7fe86b816edb423498c5dd937c81279c: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "7fe86b816edb423498c5dd937c81279c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fe86b816edb423498c5dd937c81279c" member_type: VOTER last_known_addr { host: "127.11.102.1" port: 33553 } } peers { permanent_uuid: "4fe4e45f6e02471594ff656d8aff63ac" member_type: VOTER last_known_addr { host: "127.11.102.2" port: 38643 } } peers { permanent_uuid: "2316ce59fd744c01a82cb798c635b913" member_type: VOTER last_known_addr { host: "127.11.102.3" port: 37845 } } }
I20260620 06:07:21.030073 19739 tablet_replica.cc:442] T 6995327a20334d50bfcdc8b70ffb4a28 P 7fe86b816edb423498c5dd937c81279c: This TxnStatusTablet replica's current role is: LEADER
I20260620 06:07:21.030787 19702 tablet_replica.cc:439] T 6995327a20334d50bfcdc8b70ffb4a28 P 4fe4e45f6e02471594ff656d8aff63ac: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "7fe86b816edb423498c5dd937c81279c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fe86b816edb423498c5dd937c81279c" member_type: VOTER last_known_addr { host: "127.11.102.1" port: 33553 } } peers { permanent_uuid: "4fe4e45f6e02471594ff656d8aff63ac" member_type: VOTER last_known_addr { host: "127.11.102.2" port: 38643 } } peers { permanent_uuid: "2316ce59fd744c01a82cb798c635b913" member_type: VOTER last_known_addr { host: "127.11.102.3" port: 37845 } } }
I20260620 06:07:21.030893 19702 tablet_replica.cc:442] T 6995327a20334d50bfcdc8b70ffb4a28 P 4fe4e45f6e02471594ff656d8aff63ac: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 06:07:21.030944 19701 tablet_replica.cc:439] T 6995327a20334d50bfcdc8b70ffb4a28 P 2316ce59fd744c01a82cb798c635b913: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "7fe86b816edb423498c5dd937c81279c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fe86b816edb423498c5dd937c81279c" member_type: VOTER last_known_addr { host: "127.11.102.1" port: 33553 } } peers { permanent_uuid: "4fe4e45f6e02471594ff656d8aff63ac" member_type: VOTER last_known_addr { host: "127.11.102.2" port: 38643 } } peers { permanent_uuid: "2316ce59fd744c01a82cb798c635b913" member_type: VOTER last_known_addr { host: "127.11.102.3" port: 37845 } } }
I20260620 06:07:21.031054 19701 tablet_replica.cc:442] T 6995327a20334d50bfcdc8b70ffb4a28 P 2316ce59fd744c01a82cb798c635b913: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 06:07:21.038657 19738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:07:21.034580 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57400 (local address 127.11.102.2:38643)
0620 06:07:21.034679 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0620 06:07:21.034683 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 06:07:21.034876 (+ 193us) server_negotiation.cc:407] Connection header received
0620 06:07:21.035085 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:07:21.035089 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:07:21.035146 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:07:21.035218 (+ 72us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 06:07:21.035579 (+ 361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:07:21.036136 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:07:21.037320 (+ 1184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:07:21.037522 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:07:21.037659 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:07:21.037894 (+ 235us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 06:07:21.038284 (+ 390us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 06:07:21.038439 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0620 06:07:21.038514 (+ 75us) server_negotiation.cc:299] Negotiation successful
0620 06:07:21.038560 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260620 06:07:21.048624 19692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:07:21.045125 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.102.2:59335 (local address 127.11.102.1:33553)
0620 06:07:21.045252 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0620 06:07:21.045255 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 06:07:21.045372 (+ 117us) server_negotiation.cc:407] Connection header received
0620 06:07:21.045501 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:07:21.045503 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:07:21.045539 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:07:21.045606 (+ 67us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 06:07:21.045870 (+ 264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:07:21.046344 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:07:21.047561 (+ 1217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:07:21.047848 (+ 287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:07:21.047938 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:07:21.048253 (+ 315us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 06:07:21.048313 (+ 60us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 06:07:21.048363 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0620 06:07:21.048479 (+ 116us) server_negotiation.cc:299] Negotiation successful
0620 06:07:21.048518 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260620 06:07:21.048722 19756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:07:21.044995 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.102.1:33553 (local address 127.11.102.2:59335)
0620 06:07:21.045306 (+ 311us) negotiation.cc:107] Waiting for socket to connect
0620 06:07:21.045320 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 06:07:21.045417 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:07:21.045640 (+ 223us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:07:21.045643 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:07:21.045651 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 06:07:21.045773 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:07:21.045779 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:07:21.046446 (+ 667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:07:21.046449 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:07:21.047424 (+ 975us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:07:21.047434 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:07:21.047978 (+ 544us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:07:21.047982 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:07:21.048072 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:07:21.048121 (+ 49us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 06:07:21.048394 (+ 273us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 06:07:21.048401 (+ 7us) client_negotiation.cc:769] Sending connection context
0620 06:07:21.048474 (+ 73us) client_negotiation.cc:240] Negotiation successful
0620 06:07:21.048561 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":210,"thread_start_us":121,"threads_started":1}
I20260620 06:07:21.055493 19621 raft_consensus.cc:1261] T 09d9ebc901d1409db79307028aa69ffa P 2316ce59fd744c01a82cb798c635b913 [term 1 FOLLOWER]: Refusing update from remote peer 4fe4e45f6e02471594ff656d8aff63ac: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 06:07:21.055497 19341 raft_consensus.cc:1261] T 09d9ebc901d1409db79307028aa69ffa P 7fe86b816edb423498c5dd937c81279c [term 1 FOLLOWER]: Refusing update from remote peer 4fe4e45f6e02471594ff656d8aff63ac: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 06:07:21.056216 19725 consensus_queue.cc:1035] T 09d9ebc901d1409db79307028aa69ffa P 4fe4e45f6e02471594ff656d8aff63ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "2316ce59fd744c01a82cb798c635b913" member_type: VOTER last_known_addr { host: "127.11.102.3" port: 37845 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 06:07:21.056579 19702 consensus_queue.cc:1035] T 09d9ebc901d1409db79307028aa69ffa P 4fe4e45f6e02471594ff656d8aff63ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "7fe86b816edb423498c5dd937c81279c" member_type: VOTER last_known_addr { host: "127.11.102.1" port: 33553 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 06:07:21.077201 19738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:07:21.073577 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.102.1:45057 (local address 127.11.102.2:38643)
0620 06:07:21.073690 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0620 06:07:21.073695 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 06:07:21.073825 (+ 130us) server_negotiation.cc:407] Connection header received
0620 06:07:21.073908 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:07:21.073911 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:07:21.073958 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:07:21.074037 (+ 79us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 06:07:21.074271 (+ 234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:07:21.075050 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:07:21.076108 (+ 1058us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:07:21.076449 (+ 341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:07:21.076543 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:07:21.076799 (+ 256us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 06:07:21.076874 (+ 75us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 06:07:21.076942 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0620 06:07:21.077014 (+ 72us) server_negotiation.cc:299] Negotiation successful
0620 06:07:21.077058 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260620 06:07:21.077488 19762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:07:21.073462 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.102.2:38643 (local address 127.11.102.1:45057)
0620 06:07:21.073743 (+ 281us) negotiation.cc:107] Waiting for socket to connect
0620 06:07:21.073756 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 06:07:21.073841 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:07:21.074048 (+ 207us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:07:21.074050 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:07:21.074055 (+ 5us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 06:07:21.074177 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:07:21.074181 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:07:21.075140 (+ 959us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:07:21.075145 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:07:21.076003 (+ 858us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:07:21.076013 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:07:21.076558 (+ 545us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:07:21.076561 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:07:21.076638 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:07:21.076680 (+ 42us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 06:07:21.076953 (+ 273us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 06:07:21.076958 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 06:07:21.077006 (+ 48us) client_negotiation.cc:240] Negotiation successful
0620 06:07:21.077066 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":109,"threads_started":1}
I20260620 06:07:21.106190 19184 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56276:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 06:07:21.106366 19184 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56276:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 06:07:21.108661 19184 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 05824e2e98474e8da81f776cc916c2e8: Sending DeleteTablet for 3 replicas of tablet 09d9ebc901d1409db79307028aa69ffa
I20260620 06:07:21.109239 19461 tablet_service.cc:1514] Processing DeleteTablet for tablet 09d9ebc901d1409db79307028aa69ffa with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 06:07:21 UTC) from {username='oryx', principal='oryx/127.11.102.62@KRBTEST.COM'} at 127.0.0.1:57384
I20260620 06:07:21.109385 19321 tablet_service.cc:1514] Processing DeleteTablet for tablet 09d9ebc901d1409db79307028aa69ffa with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 06:07:21 UTC) from {username='oryx', principal='oryx/127.11.102.62@KRBTEST.COM'} at 127.0.0.1:52890
I20260620 06:07:21.109686 19601 tablet_service.cc:1514] Processing DeleteTablet for tablet 09d9ebc901d1409db79307028aa69ffa with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 06:07:21 UTC) from {username='oryx', principal='oryx/127.11.102.62@KRBTEST.COM'} at 127.0.0.1:44116
I20260620 06:07:21.110013 19767 tablet_replica.cc:330] T 09d9ebc901d1409db79307028aa69ffa P 2316ce59fd744c01a82cb798c635b913: stopping tablet replica
I20260620 06:07:21.110194 19767 raft_consensus.cc:2229] T 09d9ebc901d1409db79307028aa69ffa P 2316ce59fd744c01a82cb798c635b913 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 06:07:21.110282 19765 tablet_replica.cc:330] T 09d9ebc901d1409db79307028aa69ffa P 4fe4e45f6e02471594ff656d8aff63ac: stopping tablet replica
I20260620 06:07:21.110450 19765 raft_consensus.cc:2229] T 09d9ebc901d1409db79307028aa69ffa P 4fe4e45f6e02471594ff656d8aff63ac [term 1 LEADER]: Raft consensus shutting down.
I20260620 06:07:21.110631 19767 raft_consensus.cc:2258] T 09d9ebc901d1409db79307028aa69ffa P 2316ce59fd744c01a82cb798c635b913 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 06:07:21.110760 19765 raft_consensus.cc:2258] T 09d9ebc901d1409db79307028aa69ffa P 4fe4e45f6e02471594ff656d8aff63ac [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 06:07:21.111539 19767 ts_tablet_manager.cc:1905] T 09d9ebc901d1409db79307028aa69ffa P 2316ce59fd744c01a82cb798c635b913: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 06:07:21.111553 19765 ts_tablet_manager.cc:1905] T 09d9ebc901d1409db79307028aa69ffa P 4fe4e45f6e02471594ff656d8aff63ac: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 06:07:21.114625 19765 ts_tablet_manager.cc:1918] T 09d9ebc901d1409db79307028aa69ffa P 4fe4e45f6e02471594ff656d8aff63ac: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 06:07:21.114708 19765 log.cc:1196] T 09d9ebc901d1409db79307028aa69ffa P 4fe4e45f6e02471594ff656d8aff63ac: Deleting WAL directory at /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781935603120653-11672-0/minicluster-data/ts-1/wal/wals/09d9ebc901d1409db79307028aa69ffa
I20260620 06:07:21.114815 19767 ts_tablet_manager.cc:1918] T 09d9ebc901d1409db79307028aa69ffa P 2316ce59fd744c01a82cb798c635b913: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 06:07:21.114949 19767 log.cc:1196] T 09d9ebc901d1409db79307028aa69ffa P 2316ce59fd744c01a82cb798c635b913: Deleting WAL directory at /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781935603120653-11672-0/minicluster-data/ts-2/wal/wals/09d9ebc901d1409db79307028aa69ffa
I20260620 06:07:21.114970 19765 ts_tablet_manager.cc:1939] T 09d9ebc901d1409db79307028aa69ffa P 4fe4e45f6e02471594ff656d8aff63ac: Deleting consensus metadata
I20260620 06:07:21.115338 19767 ts_tablet_manager.cc:1939] T 09d9ebc901d1409db79307028aa69ffa P 2316ce59fd744c01a82cb798c635b913: Deleting consensus metadata
I20260620 06:07:21.115685 19171 catalog_manager.cc:4841] TS 4fe4e45f6e02471594ff656d8aff63ac (127.11.102.2:38643): tablet 09d9ebc901d1409db79307028aa69ffa (table test-table [id=c95c167070e944d8b5faf77ed80b5c98]) successfully deleted
I20260620 06:07:21.116343 19170 catalog_manager.cc:4841] TS 2316ce59fd744c01a82cb798c635b913 (127.11.102.3:37845): tablet 09d9ebc901d1409db79307028aa69ffa (table test-table [id=c95c167070e944d8b5faf77ed80b5c98]) successfully deleted
I20260620 06:07:21.120002 19766 tablet_replica.cc:330] T 09d9ebc901d1409db79307028aa69ffa P 7fe86b816edb423498c5dd937c81279c: stopping tablet replica
I20260620 06:07:21.120190 19766 raft_consensus.cc:2229] T 09d9ebc901d1409db79307028aa69ffa P 7fe86b816edb423498c5dd937c81279c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 06:07:21.120416 19766 raft_consensus.cc:2258] T 09d9ebc901d1409db79307028aa69ffa P 7fe86b816edb423498c5dd937c81279c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 06:07:21.121161 19766 ts_tablet_manager.cc:1905] T 09d9ebc901d1409db79307028aa69ffa P 7fe86b816edb423498c5dd937c81279c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 06:07:21.124014 19766 ts_tablet_manager.cc:1918] T 09d9ebc901d1409db79307028aa69ffa P 7fe86b816edb423498c5dd937c81279c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 06:07:21.124095 19766 log.cc:1196] T 09d9ebc901d1409db79307028aa69ffa P 7fe86b816edb423498c5dd937c81279c: Deleting WAL directory at /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781935603120653-11672-0/minicluster-data/ts-0/wal/wals/09d9ebc901d1409db79307028aa69ffa
I20260620 06:07:21.124341 19766 ts_tablet_manager.cc:1939] T 09d9ebc901d1409db79307028aa69ffa P 7fe86b816edb423498c5dd937c81279c: Deleting consensus metadata
I20260620 06:07:21.125006 19169 catalog_manager.cc:4841] TS 7fe86b816edb423498c5dd937c81279c (127.11.102.1:33553): tablet 09d9ebc901d1409db79307028aa69ffa (table test-table [id=c95c167070e944d8b5faf77ed80b5c98]) successfully deleted
Jun 20 06:07:21 dist-test-slave-9hm5 krb5kdc[19139](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935641, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 20 06:07:21 dist-test-slave-9hm5 krb5kdc[19139](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.11.102.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 06:07:21 dist-test-slave-9hm5 krb5kdc[19139](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.11.102.62@KRBTEST.COM, Server not found in Kerberos database
W20260620 06:07:21.147202 19233 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 06:07:21.139268 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56284 (local address 127.11.102.62:36245)
0620 06:07:21.139385 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0620 06:07:21.139390 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 06:07:21.139656 (+ 266us) server_negotiation.cc:407] Connection header received
0620 06:07:21.139857 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:07:21.139861 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:07:21.139920 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:07:21.140017 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:07:21.140914 (+ 897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:07:21.141736 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:07:21.142940 (+ 1204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:07:21.143279 (+ 339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:07:21.143383 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:07:21.147063 (+ 3680us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56284: BlockingRecv error: recv got EOF from 127.0.0.1:56284 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
Jun 20 06:07:21 dist-test-slave-9hm5 krb5kdc[19139](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935641, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.11.102.62@KRBTEST.COM
I20260620 06:07:21.159271 19233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:07:21.149227 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56286 (local address 127.11.102.62:36245)
0620 06:07:21.149334 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0620 06:07:21.149338 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 06:07:21.149744 (+ 406us) server_negotiation.cc:407] Connection header received
0620 06:07:21.149830 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:07:21.149833 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:07:21.149878 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:07:21.149963 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:07:21.150647 (+ 684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:07:21.151408 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:07:21.152568 (+ 1160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:07:21.152930 (+ 362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:07:21.153030 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:07:21.155792 (+ 2762us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:07:21.155796 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:07:21.155799 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:07:21.155828 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:07:21.157489 (+ 1661us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:07:21.157995 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:07:21.157999 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:07:21.158001 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:07:21.158085 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:07:21.158362 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:07:21.158365 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:07:21.158367 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:07:21.158576 (+ 209us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:07:21.158698 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0620 06:07:21.158998 (+ 300us) server_negotiation.cc:299] Negotiation successful
0620 06:07:21.159111 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260620 06:07:21.162189 19184 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56286:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 06:07:21.162633 19184 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 06:07:21.167600 19321 tablet_service.cc:1467] Processing CreateTablet for tablet cf2f31db85c94bc8ab7594ab5f223272 (DEFAULT_TABLE table=test-table [id=485ec1893d5c494394d797bb97405b7e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 06:07:21.167927 19321 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cf2f31db85c94bc8ab7594ab5f223272. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 06:07:21.168066 19461 tablet_service.cc:1467] Processing CreateTablet for tablet cf2f31db85c94bc8ab7594ab5f223272 (DEFAULT_TABLE table=test-table [id=485ec1893d5c494394d797bb97405b7e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 06:07:21.168325 19461 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cf2f31db85c94bc8ab7594ab5f223272. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 06:07:21.170218 19696 tablet_bootstrap.cc:492] T cf2f31db85c94bc8ab7594ab5f223272 P 7fe86b816edb423498c5dd937c81279c: Bootstrap starting.
I20260620 06:07:21.170675 19601 tablet_service.cc:1467] Processing CreateTablet for tablet cf2f31db85c94bc8ab7594ab5f223272 (DEFAULT_TABLE table=test-table [id=485ec1893d5c494394d797bb97405b7e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 06:07:21.170921 19601 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cf2f31db85c94bc8ab7594ab5f223272. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 06:07:21.171013 19696 tablet_bootstrap.cc:654] T cf2f31db85c94bc8ab7594ab5f223272 P 7fe86b816edb423498c5dd937c81279c: Neither blocks nor log segments found. Creating new log.
I20260620 06:07:21.173135 19697 tablet_bootstrap.cc:492] T cf2f31db85c94bc8ab7594ab5f223272 P 2316ce59fd744c01a82cb798c635b913: Bootstrap starting.
I20260620 06:07:21.173183 19695 tablet_bootstrap.cc:492] T cf2f31db85c94bc8ab7594ab5f223272 P 4fe4e45f6e02471594ff656d8aff63ac: Bootstrap starting.
I20260620 06:07:21.174017 19697 tablet_bootstrap.cc:654] T cf2f31db85c94bc8ab7594ab5f223272 P 2316ce59fd744c01a82cb798c635b913: Neither blocks nor log segments found. Creating new log.
I20260620 06:07:21.174079 19695 tablet_bootstrap.cc:654] T cf2f31db85c94bc8ab7594ab5f223272 P 4fe4e45f6e02471594ff656d8aff63ac: Neither blocks nor log segments found. Creating new log.
I20260620 06:07:21.175694 19697 tablet_bootstrap.cc:492] T cf2f31db85c94bc8ab7594ab5f223272 P 2316ce59fd744c01a82cb798c635b913: No bootstrap required, opened a new log
I20260620 06:07:21.175777 19697 ts_tablet_manager.cc:1397] T cf2f31db85c94bc8ab7594ab5f223272 P 2316ce59fd744c01a82cb798c635b913: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260620 06:07:21.175980 19696 tablet_bootstrap.cc:492] T cf2f31db85c94bc8ab7594ab5f223272 P 7fe86b816edb423498c5dd937c81279c: No bootstrap required, opened a new log
I20260620 06:07:21.176051 19696 ts_tablet_manager.cc:1397] T cf2f31db85c94bc8ab7594ab5f223272 P 7fe86b816edb423498c5dd937c81279c: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.001s
I20260620 06:07:21.176115 19697 raft_consensus.cc:348] T cf2f31db85c94bc8ab7594ab5f223272 P 2316ce59fd744c01a82cb798c635b913 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fe86b816edb423498c5dd937c81279c" member_type: VOTER last_known_addr { host: "127.11.102.1" port: 33553 } } peers { permanent_uuid: "4fe4e45f6e02471594ff656d8aff63ac" member_type: VOTER last_known_addr { host: "127.11.102.2" port: 38643 } } peers { permanent_uuid: "2316ce59fd744c01a82cb798c635b913" member_type: VOTER last_known_addr { host: "127.11.102.3" port: 37845 } }
I20260620 06:07:21.176213 19697 raft_consensus.cc:374] T cf2f31db85c94bc8ab7594ab5f223272 P 2316ce59fd744c01a82cb798c635b913 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 06:07:21.176239 19697 raft_consensus.cc:729] T cf2f31db85c94bc8ab7594ab5f223272 P 2316ce59fd744c01a82cb798c635b913 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2316ce59fd744c01a82cb798c635b913, State: Initialized, Role: FOLLOWER
I20260620 06:07:21.176343 19697 consensus_queue.cc:260] T cf2f31db85c94bc8ab7594ab5f223272 P 2316ce59fd744c01a82cb798c635b913 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fe86b816edb423498c5dd937c81279c" member_type: VOTER last_known_addr { host: "127.11.102.1" port: 33553 } } peers { permanent_uuid: "4fe4e45f6e02471594ff656d8aff63ac" member_type: VOTER last_known_addr { host: "127.11.102.2" port: 38643 } } peers { permanent_uuid: "2316ce59fd744c01a82cb798c635b913" member_type: VOTER last_known_addr { host: "127.11.102.3" port: 37845 } }
I20260620 06:07:21.176388 19696 raft_consensus.cc:348] T cf2f31db85c94bc8ab7594ab5f223272 P 7fe86b816edb423498c5dd937c81279c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fe86b816edb423498c5dd937c81279c" member_type: VOTER last_known_addr { host: "127.11.102.1" port: 33553 } } peers { permanent_uuid: "4fe4e45f6e02471594ff656d8aff63ac" member_type: VOTER last_known_addr { host: "127.11.102.2" port: 38643 } } peers { permanent_uuid: "2316ce59fd744c01a82cb798c635b913" member_type: VOTER last_known_addr { host: "127.11.102.3" port: 37845 } }
I20260620 06:07:21.176482 19696 raft_consensus.cc:374] T cf2f31db85c94bc8ab7594ab5f223272 P 7fe86b816edb423498c5dd937c81279c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 06:07:21.176510 19696 raft_consensus.cc:729] T cf2f31db85c94bc8ab7594ab5f223272 P 7fe86b816edb423498c5dd937c81279c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7fe86b816edb423498c5dd937c81279c, State: Initialized, Role: FOLLOWER
I20260620 06:07:21.176555 19697 ts_tablet_manager.cc:1428] T cf2f31db85c94bc8ab7594ab5f223272 P 2316ce59fd744c01a82cb798c635b913: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260620 06:07:21.176631 19696 consensus_queue.cc:260] T cf2f31db85c94bc8ab7594ab5f223272 P 7fe86b816edb423498c5dd937c81279c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fe86b816edb423498c5dd937c81279c" member_type: VOTER last_known_addr { host: "127.11.102.1" port: 33553 } } peers { permanent_uuid: "4fe4e45f6e02471594ff656d8aff63ac" member_type: VOTER last_known_addr { host: "127.11.102.2" port: 38643 } } peers { permanent_uuid: "2316ce59fd744c01a82cb798c635b913" member_type: VOTER last_known_addr { host: "127.11.102.3" port: 37845 } }
I20260620 06:07:21.176879 19696 ts_tablet_manager.cc:1428] T cf2f31db85c94bc8ab7594ab5f223272 P 7fe86b816edb423498c5dd937c81279c: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260620 06:07:21.178100 19695 tablet_bootstrap.cc:492] T cf2f31db85c94bc8ab7594ab5f223272 P 4fe4e45f6e02471594ff656d8aff63ac: No bootstrap required, opened a new log
I20260620 06:07:21.178179 19695 ts_tablet_manager.cc:1397] T cf2f31db85c94bc8ab7594ab5f223272 P 4fe4e45f6e02471594ff656d8aff63ac: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.001s
I20260620 06:07:21.178601 19695 raft_consensus.cc:348] T cf2f31db85c94bc8ab7594ab5f223272 P 4fe4e45f6e02471594ff656d8aff63ac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fe86b816edb423498c5dd937c81279c" member_type: VOTER last_known_addr { host: "127.11.102.1" port: 33553 } } peers { permanent_uuid: "4fe4e45f6e02471594ff656d8aff63ac" member_type: VOTER last_known_addr { host: "127.11.102.2" port: 38643 } } peers { permanent_uuid: "2316ce59fd744c01a82cb798c635b913" member_type: VOTER last_known_addr { host: "127.11.102.3" port: 37845 } }
I20260620 06:07:21.178699 19695 raft_consensus.cc:374] T cf2f31db85c94bc8ab7594ab5f223272 P 4fe4e45f6e02471594ff656d8aff63ac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 06:07:21.178727 19695 raft_consensus.cc:729] T cf2f31db85c94bc8ab7594ab5f223272 P 4fe4e45f6e02471594ff656d8aff63ac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4fe4e45f6e02471594ff656d8aff63ac, State: Initialized, Role: FOLLOWER
I20260620 06:07:21.178853 19695 consensus_queue.cc:260] T cf2f31db85c94bc8ab7594ab5f223272 P 4fe4e45f6e02471594ff656d8aff63ac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fe86b816edb423498c5dd937c81279c" member_type: VOTER last_known_addr { host: "127.11.102.1" port: 33553 } } peers { permanent_uuid: "4fe4e45f6e02471594ff656d8aff63ac" member_type: VOTER last_known_addr { host: "127.11.102.2" port: 38643 } } peers { permanent_uuid: "2316ce59fd744c01a82cb798c635b913" member_type: VOTER last_known_addr { host: "127.11.102.3" port: 37845 } }
I20260620 06:07:21.179081 19695 ts_tablet_manager.cc:1428] T cf2f31db85c94bc8ab7594ab5f223272 P 4fe4e45f6e02471594ff656d8aff63ac: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260620 06:07:21.227246 19757 raft_consensus.cc:482] T cf2f31db85c94bc8ab7594ab5f223272 P 7fe86b816edb423498c5dd937c81279c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 06:07:21.227386 19757 raft_consensus.cc:504] T cf2f31db85c94bc8ab7594ab5f223272 P 7fe86b816edb423498c5dd937c81279c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fe86b816edb423498c5dd937c81279c" member_type: VOTER last_known_addr { host: "127.11.102.1" port: 33553 } } peers { permanent_uuid: "4fe4e45f6e02471594ff656d8aff63ac" member_type: VOTER last_known_addr { host: "127.11.102.2" port: 38643 } } peers { permanent_uuid: "2316ce59fd744c01a82cb798c635b913" member_type: VOTER last_known_addr { host: "127.11.102.3" port: 37845 } }
I20260620 06:07:21.227876 19757 leader_election.cc:288] T cf2f31db85c94bc8ab7594ab5f223272 P 7fe86b816edb423498c5dd937c81279c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4fe4e45f6e02471594ff656d8aff63ac (127.11.102.2:38643), 2316ce59fd744c01a82cb798c635b913 (127.11.102.3:37845)
I20260620 06:07:21.228252 19471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf2f31db85c94bc8ab7594ab5f223272" candidate_uuid: "7fe86b816edb423498c5dd937c81279c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4fe4e45f6e02471594ff656d8aff63ac" is_pre_election: true
I20260620 06:07:21.228273 19621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf2f31db85c94bc8ab7594ab5f223272" candidate_uuid: "7fe86b816edb423498c5dd937c81279c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2316ce59fd744c01a82cb798c635b913" is_pre_election: true
I20260620 06:07:21.228403 19621 raft_consensus.cc:2454] T cf2f31db85c94bc8ab7594ab5f223272 P 2316ce59fd744c01a82cb798c635b913 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7fe86b816edb423498c5dd937c81279c in term 0.
I20260620 06:07:21.228403 19471 raft_consensus.cc:2454] T cf2f31db85c94bc8ab7594ab5f223272 P 4fe4e45f6e02471594ff656d8aff63ac [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7fe86b816edb423498c5dd937c81279c in term 0.
I20260620 06:07:21.228708 19274 leader_election.cc:302] T cf2f31db85c94bc8ab7594ab5f223272 P 7fe86b816edb423498c5dd937c81279c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2316ce59fd744c01a82cb798c635b913, 7fe86b816edb423498c5dd937c81279c; no voters:
I20260620 06:07:21.228892 19757 raft_consensus.cc:2781] T cf2f31db85c94bc8ab7594ab5f223272 P 7fe86b816edb423498c5dd937c81279c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 06:07:21.228981 19757 raft_consensus.cc:482] T cf2f31db85c94bc8ab7594ab5f223272 P 7fe86b816edb423498c5dd937c81279c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 06:07:21.229054 19757 raft_consensus.cc:3037] T cf2f31db85c94bc8ab7594ab5f223272 P 7fe86b816edb423498c5dd937c81279c [term 0 FOLLOWER]: Advancing to term 1
I20260620 06:07:21.230053 19757 raft_consensus.cc:504] T cf2f31db85c94bc8ab7594ab5f223272 P 7fe86b816edb423498c5dd937c81279c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fe86b816edb423498c5dd937c81279c" member_type: VOTER last_known_addr { host: "127.11.102.1" port: 33553 } } peers { permanent_uuid: "4fe4e45f6e02471594ff656d8aff63ac" member_type: VOTER last_known_addr { host: "127.11.102.2" port: 38643 } } peers { permanent_uuid: "2316ce59fd744c01a82cb798c635b913" member_type: VOTER last_known_addr { host: "127.11.102.3" port: 37845 } }
I20260620 06:07:21.230406 19757 leader_election.cc:288] T cf2f31db85c94bc8ab7594ab5f223272 P 7fe86b816edb423498c5dd937c81279c [CANDIDATE]: Term 1 election: Requested vote from peers 4fe4e45f6e02471594ff656d8aff63ac (127.11.102.2:38643), 2316ce59fd744c01a82cb798c635b913 (127.11.102.3:37845)
I20260620 06:07:21.230722 19471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf2f31db85c94bc8ab7594ab5f223272" candidate_uuid: "7fe86b816edb423498c5dd937c81279c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4fe4e45f6e02471594ff656d8aff63ac"
I20260620 06:07:21.230732 19621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf2f31db85c94bc8ab7594ab5f223272" candidate_uuid: "7fe86b816edb423498c5dd937c81279c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2316ce59fd744c01a82cb798c635b913"
I20260620 06:07:21.230835 19471 raft_consensus.cc:3037] T cf2f31db85c94bc8ab7594ab5f223272 P 4fe4e45f6e02471594ff656d8aff63ac [term 0 FOLLOWER]: Advancing to term 1
I20260620 06:07:21.230859 19621 raft_consensus.cc:3037] T cf2f31db85c94bc8ab7594ab5f223272 P 2316ce59fd744c01a82cb798c635b913 [term 0 FOLLOWER]: Advancing to term 1
I20260620 06:07:21.231791 19621 raft_consensus.cc:2454] T cf2f31db85c94bc8ab7594ab5f223272 P 2316ce59fd744c01a82cb798c635b913 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7fe86b816edb423498c5dd937c81279c in term 1.
I20260620 06:07:21.231992 19471 raft_consensus.cc:2454] T cf2f31db85c94bc8ab7594ab5f223272 P 4fe4e45f6e02471594ff656d8aff63ac [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7fe86b816edb423498c5dd937c81279c in term 1.
I20260620 06:07:21.232096 19274 leader_election.cc:302] T cf2f31db85c94bc8ab7594ab5f223272 P 7fe86b816edb423498c5dd937c81279c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2316ce59fd744c01a82cb798c635b913, 7fe86b816edb423498c5dd937c81279c; no voters:
I20260620 06:07:21.232290 19757 raft_consensus.cc:2781] T cf2f31db85c94bc8ab7594ab5f223272 P 7fe86b816edb423498c5dd937c81279c [term 1 FOLLOWER]: Leader election won for term 1
I20260620 06:07:21.232378 19757 raft_consensus.cc:686] T cf2f31db85c94bc8ab7594ab5f223272 P 7fe86b816edb423498c5dd937c81279c [term 1 LEADER]: Becoming Leader. State: Replica: 7fe86b816edb423498c5dd937c81279c, State: Running, Role: LEADER
I20260620 06:07:21.232528 19757 consensus_queue.cc:237] T cf2f31db85c94bc8ab7594ab5f223272 P 7fe86b816edb423498c5dd937c81279c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fe86b816edb423498c5dd937c81279c" member_type: VOTER last_known_addr { host: "127.11.102.1" port: 33553 } } peers { permanent_uuid: "4fe4e45f6e02471594ff656d8aff63ac" member_type: VOTER last_known_addr { host: "127.11.102.2" port: 38643 } } peers { permanent_uuid: "2316ce59fd744c01a82cb798c635b913" member_type: VOTER last_known_addr { host: "127.11.102.3" port: 37845 } }
I20260620 06:07:21.234246 19184 catalog_manager.cc:5495] T cf2f31db85c94bc8ab7594ab5f223272 P 7fe86b816edb423498c5dd937c81279c reported cstate change: term changed from 0 to 1, leader changed from <none> to 7fe86b816edb423498c5dd937c81279c (127.11.102.1). New cstate: current_term: 1 leader_uuid: "7fe86b816edb423498c5dd937c81279c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fe86b816edb423498c5dd937c81279c" member_type: VOTER last_known_addr { host: "127.11.102.1" port: 33553 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4fe4e45f6e02471594ff656d8aff63ac" member_type: VOTER last_known_addr { host: "127.11.102.2" port: 38643 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2316ce59fd744c01a82cb798c635b913" member_type: VOTER last_known_addr { host: "127.11.102.3" port: 37845 } health_report { overall_health: UNKNOWN } } }
I20260620 06:07:21.255998 19692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:07:21.252409 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52922 (local address 127.11.102.1:33553)
0620 06:07:21.252563 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0620 06:07:21.252567 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 06:07:21.252585 (+ 18us) server_negotiation.cc:407] Connection header received
0620 06:07:21.252718 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:07:21.252721 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:07:21.252774 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:07:21.252861 (+ 87us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 06:07:21.253113 (+ 252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:07:21.253875 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:07:21.254769 (+ 894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:07:21.255107 (+ 338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:07:21.255217 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:07:21.255457 (+ 240us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 06:07:21.255558 (+ 101us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 06:07:21.255680 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0620 06:07:21.255798 (+ 118us) server_negotiation.cc:299] Negotiation successful
0620 06:07:21.255852 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260620 06:07:21.258611 19621 raft_consensus.cc:1261] T cf2f31db85c94bc8ab7594ab5f223272 P 2316ce59fd744c01a82cb798c635b913 [term 1 FOLLOWER]: Refusing update from remote peer 7fe86b816edb423498c5dd937c81279c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 06:07:21.258608 19471 raft_consensus.cc:1261] T cf2f31db85c94bc8ab7594ab5f223272 P 4fe4e45f6e02471594ff656d8aff63ac [term 1 FOLLOWER]: Refusing update from remote peer 7fe86b816edb423498c5dd937c81279c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 06:07:21.259066 19739 consensus_queue.cc:1035] T cf2f31db85c94bc8ab7594ab5f223272 P 7fe86b816edb423498c5dd937c81279c [LEADER]: Connected to new peer: Peer: permanent_uuid: "2316ce59fd744c01a82cb798c635b913" member_type: VOTER last_known_addr { host: "127.11.102.3" port: 37845 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 06:07:21.259250 19757 consensus_queue.cc:1035] T cf2f31db85c94bc8ab7594ab5f223272 P 7fe86b816edb423498c5dd937c81279c [LEADER]: Connected to new peer: Peer: permanent_uuid: "4fe4e45f6e02471594ff656d8aff63ac" member_type: VOTER last_known_addr { host: "127.11.102.2" port: 38643 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 06:07:21.276657 19184 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56286:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 06:07:21.276813 19184 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56286:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 06:07:21.279032 19184 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 05824e2e98474e8da81f776cc916c2e8: Sending DeleteTablet for 3 replicas of tablet cf2f31db85c94bc8ab7594ab5f223272
I20260620 06:07:21.280085 19601 tablet_service.cc:1514] Processing DeleteTablet for tablet cf2f31db85c94bc8ab7594ab5f223272 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 06:07:21 UTC) from {username='oryx', principal='oryx/127.11.102.62@KRBTEST.COM'} at 127.0.0.1:44116
I20260620 06:07:21.280296 19321 tablet_service.cc:1514] Processing DeleteTablet for tablet cf2f31db85c94bc8ab7594ab5f223272 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 06:07:21 UTC) from {username='oryx', principal='oryx/127.11.102.62@KRBTEST.COM'} at 127.0.0.1:52890
I20260620 06:07:21.280491 19461 tablet_service.cc:1514] Processing DeleteTablet for tablet cf2f31db85c94bc8ab7594ab5f223272 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 06:07:21 UTC) from {username='oryx', principal='oryx/127.11.102.62@KRBTEST.COM'} at 127.0.0.1:57384
I20260620 06:07:21.280675 19767 tablet_replica.cc:330] T cf2f31db85c94bc8ab7594ab5f223272 P 2316ce59fd744c01a82cb798c635b913: stopping tablet replica
I20260620 06:07:21.280795 19767 raft_consensus.cc:2229] T cf2f31db85c94bc8ab7594ab5f223272 P 2316ce59fd744c01a82cb798c635b913 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 06:07:21.280920 19767 raft_consensus.cc:2258] T cf2f31db85c94bc8ab7594ab5f223272 P 2316ce59fd744c01a82cb798c635b913 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 06:07:21.281077 19766 tablet_replica.cc:330] T cf2f31db85c94bc8ab7594ab5f223272 P 7fe86b816edb423498c5dd937c81279c: stopping tablet replica
I20260620 06:07:21.281211 19766 raft_consensus.cc:2229] T cf2f31db85c94bc8ab7594ab5f223272 P 7fe86b816edb423498c5dd937c81279c [term 1 LEADER]: Raft consensus shutting down.
I20260620 06:07:21.281525 19766 raft_consensus.cc:2258] T cf2f31db85c94bc8ab7594ab5f223272 P 7fe86b816edb423498c5dd937c81279c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 06:07:21.281666 19767 ts_tablet_manager.cc:1905] T cf2f31db85c94bc8ab7594ab5f223272 P 2316ce59fd744c01a82cb798c635b913: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 06:07:21.281718 19765 tablet_replica.cc:330] T cf2f31db85c94bc8ab7594ab5f223272 P 4fe4e45f6e02471594ff656d8aff63ac: stopping tablet replica
I20260620 06:07:21.281831 19765 raft_consensus.cc:2229] T cf2f31db85c94bc8ab7594ab5f223272 P 4fe4e45f6e02471594ff656d8aff63ac [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 06:07:21.281962 19765 raft_consensus.cc:2258] T cf2f31db85c94bc8ab7594ab5f223272 P 4fe4e45f6e02471594ff656d8aff63ac [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 06:07:21.282814 19766 ts_tablet_manager.cc:1905] T cf2f31db85c94bc8ab7594ab5f223272 P 7fe86b816edb423498c5dd937c81279c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 06:07:21.284220 19767 ts_tablet_manager.cc:1918] T cf2f31db85c94bc8ab7594ab5f223272 P 2316ce59fd744c01a82cb798c635b913: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 06:07:21.284304 19767 log.cc:1196] T cf2f31db85c94bc8ab7594ab5f223272 P 2316ce59fd744c01a82cb798c635b913: Deleting WAL directory at /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781935603120653-11672-0/minicluster-data/ts-2/wal/wals/cf2f31db85c94bc8ab7594ab5f223272
I20260620 06:07:21.284562 19767 ts_tablet_manager.cc:1939] T cf2f31db85c94bc8ab7594ab5f223272 P 2316ce59fd744c01a82cb798c635b913: Deleting consensus metadata
I20260620 06:07:21.285280 19170 catalog_manager.cc:4841] TS 2316ce59fd744c01a82cb798c635b913 (127.11.102.3:37845): tablet cf2f31db85c94bc8ab7594ab5f223272 (table test-table [id=485ec1893d5c494394d797bb97405b7e]) successfully deleted
I20260620 06:07:21.286347 19182 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56286:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 06:07:21.286765 19182 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 06:07:21.288810 19766 ts_tablet_manager.cc:1918] T cf2f31db85c94bc8ab7594ab5f223272 P 7fe86b816edb423498c5dd937c81279c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 06:07:21.288897 19766 log.cc:1196] T cf2f31db85c94bc8ab7594ab5f223272 P 7fe86b816edb423498c5dd937c81279c: Deleting WAL directory at /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781935603120653-11672-0/minicluster-data/ts-0/wal/wals/cf2f31db85c94bc8ab7594ab5f223272
I20260620 06:07:21.289175 19766 ts_tablet_manager.cc:1939] T cf2f31db85c94bc8ab7594ab5f223272 P 7fe86b816edb423498c5dd937c81279c: Deleting consensus metadata
I20260620 06:07:21.289862 19169 catalog_manager.cc:4841] TS 7fe86b816edb423498c5dd937c81279c (127.11.102.1:33553): tablet cf2f31db85c94bc8ab7594ab5f223272 (table test-table [id=485ec1893d5c494394d797bb97405b7e]) successfully deleted
I20260620 06:07:21.290432 19765 ts_tablet_manager.cc:1905] T cf2f31db85c94bc8ab7594ab5f223272 P 4fe4e45f6e02471594ff656d8aff63ac: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 06:07:21.292744 19765 ts_tablet_manager.cc:1918] T cf2f31db85c94bc8ab7594ab5f223272 P 4fe4e45f6e02471594ff656d8aff63ac: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 06:07:21.292824 19765 log.cc:1196] T cf2f31db85c94bc8ab7594ab5f223272 P 4fe4e45f6e02471594ff656d8aff63ac: Deleting WAL directory at /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781935603120653-11672-0/minicluster-data/ts-1/wal/wals/cf2f31db85c94bc8ab7594ab5f223272
I20260620 06:07:21.293092 19765 ts_tablet_manager.cc:1939] T cf2f31db85c94bc8ab7594ab5f223272 P 4fe4e45f6e02471594ff656d8aff63ac: Deleting consensus metadata
I20260620 06:07:21.293692 19171 catalog_manager.cc:4841] TS 4fe4e45f6e02471594ff656d8aff63ac (127.11.102.2:38643): tablet cf2f31db85c94bc8ab7594ab5f223272 (table test-table [id=485ec1893d5c494394d797bb97405b7e]) successfully deleted
I20260620 06:07:21.293792 19461 tablet_service.cc:1467] Processing CreateTablet for tablet 7ce2ad9bb8074ada92744f6b2c040abb (DEFAULT_TABLE table=test-table [id=5d7e6e0d0c49402dbd30337b9ae6e534]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 06:07:21.294040 19461 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7ce2ad9bb8074ada92744f6b2c040abb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 06:07:21.295984 19695 tablet_bootstrap.cc:492] T 7ce2ad9bb8074ada92744f6b2c040abb P 4fe4e45f6e02471594ff656d8aff63ac: Bootstrap starting.
I20260620 06:07:21.296741 19695 tablet_bootstrap.cc:654] T 7ce2ad9bb8074ada92744f6b2c040abb P 4fe4e45f6e02471594ff656d8aff63ac: Neither blocks nor log segments found. Creating new log.
I20260620 06:07:21.297294 19321 tablet_service.cc:1467] Processing CreateTablet for tablet 7ce2ad9bb8074ada92744f6b2c040abb (DEFAULT_TABLE table=test-table [id=5d7e6e0d0c49402dbd30337b9ae6e534]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 06:07:21.297528 19321 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7ce2ad9bb8074ada92744f6b2c040abb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 06:07:21.297676 19695 tablet_bootstrap.cc:492] T 7ce2ad9bb8074ada92744f6b2c040abb P 4fe4e45f6e02471594ff656d8aff63ac: No bootstrap required, opened a new log
I20260620 06:07:21.297744 19695 ts_tablet_manager.cc:1397] T 7ce2ad9bb8074ada92744f6b2c040abb P 4fe4e45f6e02471594ff656d8aff63ac: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.001s
I20260620 06:07:21.298079 19695 raft_consensus.cc:348] T 7ce2ad9bb8074ada92744f6b2c040abb P 4fe4e45f6e02471594ff656d8aff63ac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fe86b816edb423498c5dd937c81279c" member_type: VOTER last_known_addr { host: "127.11.102.1" port: 33553 } } peers { permanent_uuid: "4fe4e45f6e02471594ff656d8aff63ac" member_type: VOTER last_known_addr { host: "127.11.102.2" port: 38643 } } peers { permanent_uuid: "2316ce59fd744c01a82cb798c635b913" member_type: VOTER last_known_addr { host: "127.11.102.3" port: 37845 } }
I20260620 06:07:21.298179 19695 raft_consensus.cc:374] T 7ce2ad9bb8074ada92744f6b2c040abb P 4fe4e45f6e02471594ff656d8aff63ac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 06:07:21.298207 19695 raft_consensus.cc:729] T 7ce2ad9bb8074ada92744f6b2c040abb P 4fe4e45f6e02471594ff656d8aff63ac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4fe4e45f6e02471594ff656d8aff63ac, State: Initialized, Role: FOLLOWER
I20260620 06:07:21.298311 19695 consensus_queue.cc:260] T 7ce2ad9bb8074ada92744f6b2c040abb P 4fe4e45f6e02471594ff656d8aff63ac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fe86b816edb423498c5dd937c81279c" member_type: VOTER last_known_addr { host: "127.11.102.1" port: 33553 } } peers { permanent_uuid: "4fe4e45f6e02471594ff656d8aff63ac" member_type: VOTER last_known_addr { host: "127.11.102.2" port: 38643 } } peers { permanent_uuid: "2316ce59fd744c01a82cb798c635b913" member_type: VOTER last_known_addr { host: "127.11.102.3" port: 37845 } }
I20260620 06:07:21.298539 19695 ts_tablet_manager.cc:1428] T 7ce2ad9bb8074ada92744f6b2c040abb P 4fe4e45f6e02471594ff656d8aff63ac: Time spent starting tablet: real 0.001s user 0.001s sys 0.001s
I20260620 06:07:21.299682 19696 tablet_bootstrap.cc:492] T 7ce2ad9bb8074ada92744f6b2c040abb P 7fe86b816edb423498c5dd937c81279c: Bootstrap starting.
I20260620 06:07:21.300172 19601 tablet_service.cc:1467] Processing CreateTablet for tablet 7ce2ad9bb8074ada92744f6b2c040abb (DEFAULT_TABLE table=test-table [id=5d7e6e0d0c49402dbd30337b9ae6e534]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 06:07:21.300405 19601 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7ce2ad9bb8074ada92744f6b2c040abb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 06:07:21.300455 19696 tablet_bootstrap.cc:654] T 7ce2ad9bb8074ada92744f6b2c040abb P 7fe86b816edb423498c5dd937c81279c: Neither blocks nor log segments found. Creating new log.
I20260620 06:07:21.301414 19696 tablet_bootstrap.cc:492] T 7ce2ad9bb8074ada92744f6b2c040abb P 7fe86b816edb423498c5dd937c81279c: No bootstrap required, opened a new log
I20260620 06:07:21.301467 19696 ts_tablet_manager.cc:1397] T 7ce2ad9bb8074ada92744f6b2c040abb P 7fe86b816edb423498c5dd937c81279c: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 06:07:21.301738 19696 raft_consensus.cc:348] T 7ce2ad9bb8074ada92744f6b2c040abb P 7fe86b816edb423498c5dd937c81279c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fe86b816edb423498c5dd937c81279c" member_type: VOTER last_known_addr { host: "127.11.102.1" port: 33553 } } peers { permanent_uuid: "4fe4e45f6e02471594ff656d8aff63ac" member_type: VOTER last_known_addr { host: "127.11.102.2" port: 38643 } } peers { permanent_uuid: "2316ce59fd744c01a82cb798c635b913" member_type: VOTER last_known_addr { host: "127.11.102.3" port: 37845 } }
I20260620 06:07:21.301811 19696 raft_consensus.cc:374] T 7ce2ad9bb8074ada92744f6b2c040abb P 7fe86b816edb423498c5dd937c81279c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 06:07:21.301831 19696 raft_consensus.cc:729] T 7ce2ad9bb8074ada92744f6b2c040abb P 7fe86b816edb423498c5dd937c81279c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7fe86b816edb423498c5dd937c81279c, State: Initialized, Role: FOLLOWER
I20260620 06:07:21.301961 19696 consensus_queue.cc:260] T 7ce2ad9bb8074ada92744f6b2c040abb P 7fe86b816edb423498c5dd937c81279c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fe86b816edb423498c5dd937c81279c" member_type: VOTER last_known_addr { host: "127.11.102.1" port: 33553 } } peers { permanent_uuid: "4fe4e45f6e02471594ff656d8aff63ac" member_type: VOTER last_known_addr { host: "127.11.102.2" port: 38643 } } peers { permanent_uuid: "2316ce59fd744c01a82cb798c635b913" member_type: VOTER last_known_addr { host: "127.11.102.3" port: 37845 } }
I20260620 06:07:21.302222 19696 ts_tablet_manager.cc:1428] T 7ce2ad9bb8074ada92744f6b2c040abb P 7fe86b816edb423498c5dd937c81279c: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260620 06:07:21.302531 19697 tablet_bootstrap.cc:492] T 7ce2ad9bb8074ada92744f6b2c040abb P 2316ce59fd744c01a82cb798c635b913: Bootstrap starting.
I20260620 06:07:21.303296 19697 tablet_bootstrap.cc:654] T 7ce2ad9bb8074ada92744f6b2c040abb P 2316ce59fd744c01a82cb798c635b913: Neither blocks nor log segments found. Creating new log.
I20260620 06:07:21.304448 19697 tablet_bootstrap.cc:492] T 7ce2ad9bb8074ada92744f6b2c040abb P 2316ce59fd744c01a82cb798c635b913: No bootstrap required, opened a new log
I20260620 06:07:21.304531 19697 ts_tablet_manager.cc:1397] T 7ce2ad9bb8074ada92744f6b2c040abb P 2316ce59fd744c01a82cb798c635b913: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 06:07:21.304909 19697 raft_consensus.cc:348] T 7ce2ad9bb8074ada92744f6b2c040abb P 2316ce59fd744c01a82cb798c635b913 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fe86b816edb423498c5dd937c81279c" member_type: VOTER last_known_addr { host: "127.11.102.1" port: 33553 } } peers { permanent_uuid: "4fe4e45f6e02471594ff656d8aff63ac" member_type: VOTER last_known_addr { host: "127.11.102.2" port: 38643 } } peers { permanent_uuid: "2316ce59fd744c01a82cb798c635b913" member_type: VOTER last_known_addr { host: "127.11.102.3" port: 37845 } }
I20260620 06:07:21.305032 19697 raft_consensus.cc:374] T 7ce2ad9bb8074ada92744f6b2c040abb P 2316ce59fd744c01a82cb798c635b913 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 06:07:21.305073 19697 raft_consensus.cc:729] T 7ce2ad9bb8074ada92744f6b2c040abb P 2316ce59fd744c01a82cb798c635b913 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2316ce59fd744c01a82cb798c635b913, State: Initialized, Role: FOLLOWER
I20260620 06:07:21.305218 19697 consensus_queue.cc:260] T 7ce2ad9bb8074ada92744f6b2c040abb P 2316ce59fd744c01a82cb798c635b913 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fe86b816edb423498c5dd937c81279c" member_type: VOTER last_known_addr { host: "127.11.102.1" port: 33553 } } peers { permanent_uuid: "4fe4e45f6e02471594ff656d8aff63ac" member_type: VOTER last_known_addr { host: "127.11.102.2" port: 38643 } } peers { permanent_uuid: "2316ce59fd744c01a82cb798c635b913" member_type: VOTER last_known_addr { host: "127.11.102.3" port: 37845 } }
I20260620 06:07:21.305508 19697 ts_tablet_manager.cc:1428] T 7ce2ad9bb8074ada92744f6b2c040abb P 2316ce59fd744c01a82cb798c635b913: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260620 06:07:21.444831 19702 raft_consensus.cc:482] T 7ce2ad9bb8074ada92744f6b2c040abb P 4fe4e45f6e02471594ff656d8aff63ac [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 06:07:21.445036 19702 raft_consensus.cc:504] T 7ce2ad9bb8074ada92744f6b2c040abb P 4fe4e45f6e02471594ff656d8aff63ac [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fe86b816edb423498c5dd937c81279c" member_type: VOTER last_known_addr { host: "127.11.102.1" port: 33553 } } peers { permanent_uuid: "4fe4e45f6e02471594ff656d8aff63ac" member_type: VOTER last_known_addr { host: "127.11.102.2" port: 38643 } } peers { permanent_uuid: "2316ce59fd744c01a82cb798c635b913" member_type: VOTER last_known_addr { host: "127.11.102.3" port: 37845 } }
I20260620 06:07:21.445511 19702 leader_election.cc:288] T 7ce2ad9bb8074ada92744f6b2c040abb P 4fe4e45f6e02471594ff656d8aff63ac [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7fe86b816edb423498c5dd937c81279c (127.11.102.1:33553), 2316ce59fd744c01a82cb798c635b913 (127.11.102.3:37845)
I20260620 06:07:21.445998 19621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ce2ad9bb8074ada92744f6b2c040abb" candidate_uuid: "4fe4e45f6e02471594ff656d8aff63ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2316ce59fd744c01a82cb798c635b913" is_pre_election: true
I20260620 06:07:21.446018 19341 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ce2ad9bb8074ada92744f6b2c040abb" candidate_uuid: "4fe4e45f6e02471594ff656d8aff63ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fe86b816edb423498c5dd937c81279c" is_pre_election: true
I20260620 06:07:21.446149 19621 raft_consensus.cc:2454] T 7ce2ad9bb8074ada92744f6b2c040abb P 2316ce59fd744c01a82cb798c635b913 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4fe4e45f6e02471594ff656d8aff63ac in term 0.
I20260620 06:07:21.446230 19341 raft_consensus.cc:2454] T 7ce2ad9bb8074ada92744f6b2c040abb P 7fe86b816edb423498c5dd937c81279c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4fe4e45f6e02471594ff656d8aff63ac in term 0.
I20260620 06:07:21.446506 19414 leader_election.cc:302] T 7ce2ad9bb8074ada92744f6b2c040abb P 4fe4e45f6e02471594ff656d8aff63ac [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2316ce59fd744c01a82cb798c635b913, 4fe4e45f6e02471594ff656d8aff63ac; no voters:
I20260620 06:07:21.446699 19702 raft_consensus.cc:2781] T 7ce2ad9bb8074ada92744f6b2c040abb P 4fe4e45f6e02471594ff656d8aff63ac [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 06:07:21.446764 19702 raft_consensus.cc:482] T 7ce2ad9bb8074ada92744f6b2c040abb P 4fe4e45f6e02471594ff656d8aff63ac [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 06:07:21.446792 19702 raft_consensus.cc:3037] T 7ce2ad9bb8074ada92744f6b2c040abb P 4fe4e45f6e02471594ff656d8aff63ac [term 0 FOLLOWER]: Advancing to term 1
I20260620 06:07:21.447817 19702 raft_consensus.cc:504] T 7ce2ad9bb8074ada92744f6b2c040abb P 4fe4e45f6e02471594ff656d8aff63ac [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fe86b816edb423498c5dd937c81279c" member_type: VOTER last_known_addr { host: "127.11.102.1" port: 33553 } } peers { permanent_uuid: "4fe4e45f6e02471594ff656d8aff63ac" member_type: VOTER last_known_addr { host: "127.11.102.2" port: 38643 } } peers { permanent_uuid: "2316ce59fd744c01a82cb798c635b913" member_type: VOTER last_known_addr { host: "127.11.102.3" port: 37845 } }
I20260620 06:07:21.448201 19702 leader_election.cc:288] T 7ce2ad9bb8074ada92744f6b2c040abb P 4fe4e45f6e02471594ff656d8aff63ac [CANDIDATE]: Term 1 election: Requested vote from peers 7fe86b816edb423498c5dd937c81279c (127.11.102.1:33553), 2316ce59fd744c01a82cb798c635b913 (127.11.102.3:37845)
I20260620 06:07:21.448500 19341 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ce2ad9bb8074ada92744f6b2c040abb" candidate_uuid: "4fe4e45f6e02471594ff656d8aff63ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fe86b816edb423498c5dd937c81279c"
I20260620 06:07:21.448616 19341 raft_consensus.cc:3037] T 7ce2ad9bb8074ada92744f6b2c040abb P 7fe86b816edb423498c5dd937c81279c [term 0 FOLLOWER]: Advancing to term 1
I20260620 06:07:21.448622 19621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ce2ad9bb8074ada92744f6b2c040abb" candidate_uuid: "4fe4e45f6e02471594ff656d8aff63ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2316ce59fd744c01a82cb798c635b913"
I20260620 06:07:21.448755 19621 raft_consensus.cc:3037] T 7ce2ad9bb8074ada92744f6b2c040abb P 2316ce59fd744c01a82cb798c635b913 [term 0 FOLLOWER]: Advancing to term 1
I20260620 06:07:21.449456 19341 raft_consensus.cc:2454] T 7ce2ad9bb8074ada92744f6b2c040abb P 7fe86b816edb423498c5dd937c81279c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4fe4e45f6e02471594ff656d8aff63ac in term 1.
I20260620 06:07:21.449712 19621 raft_consensus.cc:2454] T 7ce2ad9bb8074ada92744f6b2c040abb P 2316ce59fd744c01a82cb798c635b913 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4fe4e45f6e02471594ff656d8aff63ac in term 1.
I20260620 06:07:21.449716 19413 leader_election.cc:302] T 7ce2ad9bb8074ada92744f6b2c040abb P 4fe4e45f6e02471594ff656d8aff63ac [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4fe4e45f6e02471594ff656d8aff63ac, 7fe86b816edb423498c5dd937c81279c; no voters:
I20260620 06:07:21.449880 19702 raft_consensus.cc:2781] T 7ce2ad9bb8074ada92744f6b2c040abb P 4fe4e45f6e02471594ff656d8aff63ac [term 1 FOLLOWER]: Leader election won for term 1
I20260620 06:07:21.449966 19702 raft_consensus.cc:686] T 7ce2ad9bb8074ada92744f6b2c040abb P 4fe4e45f6e02471594ff656d8aff63ac [term 1 LEADER]: Becoming Leader. State: Replica: 4fe4e45f6e02471594ff656d8aff63ac, State: Running, Role: LEADER
I20260620 06:07:21.450101 19702 consensus_queue.cc:237] T 7ce2ad9bb8074ada92744f6b2c040abb P 4fe4e45f6e02471594ff656d8aff63ac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fe86b816edb423498c5dd937c81279c" member_type: VOTER last_known_addr { host: "127.11.102.1" port: 33553 } } peers { permanent_uuid: "4fe4e45f6e02471594ff656d8aff63ac" member_type: VOTER last_known_addr { host: "127.11.102.2" port: 38643 } } peers { permanent_uuid: "2316ce59fd744c01a82cb798c635b913" member_type: VOTER last_known_addr { host: "127.11.102.3" port: 37845 } }
I20260620 06:07:21.451874 19183 catalog_manager.cc:5495] T 7ce2ad9bb8074ada92744f6b2c040abb P 4fe4e45f6e02471594ff656d8aff63ac reported cstate change: term changed from 0 to 1, leader changed from <none> to 4fe4e45f6e02471594ff656d8aff63ac (127.11.102.2). New cstate: current_term: 1 leader_uuid: "4fe4e45f6e02471594ff656d8aff63ac" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fe86b816edb423498c5dd937c81279c" member_type: VOTER last_known_addr { host: "127.11.102.1" port: 33553 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4fe4e45f6e02471594ff656d8aff63ac" member_type: VOTER last_known_addr { host: "127.11.102.2" port: 38643 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2316ce59fd744c01a82cb798c635b913" member_type: VOTER last_known_addr { host: "127.11.102.3" port: 37845 } health_report { overall_health: UNKNOWN } } }
I20260620 06:07:21.501674 19738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:07:21.498902 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57406 (local address 127.11.102.2:38643)
0620 06:07:21.499027 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0620 06:07:21.499030 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 06:07:21.499041 (+ 11us) server_negotiation.cc:407] Connection header received
0620 06:07:21.499114 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:07:21.499116 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:07:21.499153 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:07:21.499215 (+ 62us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 06:07:21.499421 (+ 206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:07:21.500037 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:07:21.500836 (+ 799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:07:21.501049 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:07:21.501125 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:07:21.501343 (+ 218us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 06:07:21.501406 (+ 63us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 06:07:21.501460 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0620 06:07:21.501538 (+ 78us) server_negotiation.cc:299] Negotiation successful
0620 06:07:21.501574 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260620 06:07:21.507138 19621 raft_consensus.cc:1261] T 7ce2ad9bb8074ada92744f6b2c040abb P 2316ce59fd744c01a82cb798c635b913 [term 1 FOLLOWER]: Refusing update from remote peer 4fe4e45f6e02471594ff656d8aff63ac: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 06:07:21.507364 19341 raft_consensus.cc:1261] T 7ce2ad9bb8074ada92744f6b2c040abb P 7fe86b816edb423498c5dd937c81279c [term 1 FOLLOWER]: Refusing update from remote peer 4fe4e45f6e02471594ff656d8aff63ac: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 06:07:21.507499 19702 consensus_queue.cc:1035] T 7ce2ad9bb8074ada92744f6b2c040abb P 4fe4e45f6e02471594ff656d8aff63ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "2316ce59fd744c01a82cb798c635b913" member_type: VOTER last_known_addr { host: "127.11.102.3" port: 37845 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 06:07:21.507827 19760 consensus_queue.cc:1035] T 7ce2ad9bb8074ada92744f6b2c040abb P 4fe4e45f6e02471594ff656d8aff63ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "7fe86b816edb423498c5dd937c81279c" member_type: VOTER last_known_addr { host: "127.11.102.1" port: 33553 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 06:07:21.538116 19184 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56286:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 06:07:21.538266 19184 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56286:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 06:07:21.540437 19184 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 05824e2e98474e8da81f776cc916c2e8: Sending DeleteTablet for 3 replicas of tablet 7ce2ad9bb8074ada92744f6b2c040abb
I20260620 06:07:21.540944 19321 tablet_service.cc:1514] Processing DeleteTablet for tablet 7ce2ad9bb8074ada92744f6b2c040abb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 06:07:21 UTC) from {username='oryx', principal='oryx/127.11.102.62@KRBTEST.COM'} at 127.0.0.1:52890
I20260620 06:07:21.540961 19461 tablet_service.cc:1514] Processing DeleteTablet for tablet 7ce2ad9bb8074ada92744f6b2c040abb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 06:07:21 UTC) from {username='oryx', principal='oryx/127.11.102.62@KRBTEST.COM'} at 127.0.0.1:57384
I20260620 06:07:21.541152 19601 tablet_service.cc:1514] Processing DeleteTablet for tablet 7ce2ad9bb8074ada92744f6b2c040abb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 06:07:21 UTC) from {username='oryx', principal='oryx/127.11.102.62@KRBTEST.COM'} at 127.0.0.1:44116
I20260620 06:07:21.541335 19765 tablet_replica.cc:330] T 7ce2ad9bb8074ada92744f6b2c040abb P 4fe4e45f6e02471594ff656d8aff63ac: stopping tablet replica
I20260620 06:07:21.541478 19765 raft_consensus.cc:2229] T 7ce2ad9bb8074ada92744f6b2c040abb P 4fe4e45f6e02471594ff656d8aff63ac [term 1 LEADER]: Raft consensus shutting down.
I20260620 06:07:21.541635 11672 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9HMp43/build/debug/bin/kudu with pid 19256
I20260620 06:07:21.541663 19765 raft_consensus.cc:2258] T 7ce2ad9bb8074ada92744f6b2c040abb P 4fe4e45f6e02471594ff656d8aff63ac [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 06:07:21.541836 19766 tablet_replica.cc:330] T 7ce2ad9bb8074ada92744f6b2c040abb P 7fe86b816edb423498c5dd937c81279c: stopping tablet replica
I20260620 06:07:21.541968 19766 raft_consensus.cc:2229] T 7ce2ad9bb8074ada92744f6b2c040abb P 7fe86b816edb423498c5dd937c81279c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 06:07:21.542058 19766 raft_consensus.cc:2258] T 7ce2ad9bb8074ada92744f6b2c040abb P 7fe86b816edb423498c5dd937c81279c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 06:07:21.542241 19767 tablet_replica.cc:330] T 7ce2ad9bb8074ada92744f6b2c040abb P 2316ce59fd744c01a82cb798c635b913: stopping tablet replica
I20260620 06:07:21.542331 19767 raft_consensus.cc:2229] T 7ce2ad9bb8074ada92744f6b2c040abb P 2316ce59fd744c01a82cb798c635b913 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 06:07:21.542404 19767 raft_consensus.cc:2258] T 7ce2ad9bb8074ada92744f6b2c040abb P 2316ce59fd744c01a82cb798c635b913 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 06:07:21.543810 19767 ts_tablet_manager.cc:1905] T 7ce2ad9bb8074ada92744f6b2c040abb P 2316ce59fd744c01a82cb798c635b913: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 06:07:21.544013 19765 ts_tablet_manager.cc:1905] T 7ce2ad9bb8074ada92744f6b2c040abb P 4fe4e45f6e02471594ff656d8aff63ac: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 06:07:21.545877 19767 ts_tablet_manager.cc:1918] T 7ce2ad9bb8074ada92744f6b2c040abb P 2316ce59fd744c01a82cb798c635b913: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 06:07:21.545970 19767 log.cc:1196] T 7ce2ad9bb8074ada92744f6b2c040abb P 2316ce59fd744c01a82cb798c635b913: Deleting WAL directory at /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781935603120653-11672-0/minicluster-data/ts-2/wal/wals/7ce2ad9bb8074ada92744f6b2c040abb
I20260620 06:07:21.546234 19767 ts_tablet_manager.cc:1939] T 7ce2ad9bb8074ada92744f6b2c040abb P 2316ce59fd744c01a82cb798c635b913: Deleting consensus metadata
I20260620 06:07:21.546876 19170 catalog_manager.cc:4841] TS 2316ce59fd744c01a82cb798c635b913 (127.11.102.3:37845): tablet 7ce2ad9bb8074ada92744f6b2c040abb (table test-table [id=5d7e6e0d0c49402dbd30337b9ae6e534]) successfully deleted
I20260620 06:07:21.547082 19765 ts_tablet_manager.cc:1918] T 7ce2ad9bb8074ada92744f6b2c040abb P 4fe4e45f6e02471594ff656d8aff63ac: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 06:07:21.547165 19765 log.cc:1196] T 7ce2ad9bb8074ada92744f6b2c040abb P 4fe4e45f6e02471594ff656d8aff63ac: Deleting WAL directory at /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781935603120653-11672-0/minicluster-data/ts-1/wal/wals/7ce2ad9bb8074ada92744f6b2c040abb
I20260620 06:07:21.547432 19765 ts_tablet_manager.cc:1939] T 7ce2ad9bb8074ada92744f6b2c040abb P 4fe4e45f6e02471594ff656d8aff63ac: Deleting consensus metadata
I20260620 06:07:21.548112 19171 catalog_manager.cc:4841] TS 4fe4e45f6e02471594ff656d8aff63ac (127.11.102.2:38643): tablet 7ce2ad9bb8074ada92744f6b2c040abb (table test-table [id=5d7e6e0d0c49402dbd30337b9ae6e534]) successfully deleted
W20260620 06:07:21.551043 19169 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.11.102.1:33553 (error 108)
I20260620 06:07:21.551266 11672 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9HMp43/build/debug/bin/kudu with pid 19396
I20260620 06:07:21.552553 19795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:07:21.552181 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.102.1:33553 (local address 127.0.0.1:52926)
0620 06:07:21.552392 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0620 06:07:21.552468 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.102.1:33553: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":75,"threads_started":1}
W20260620 06:07:21.552767 19169 catalog_manager.cc:4584] TS 7fe86b816edb423498c5dd937c81279c (127.11.102.1:33553): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 7ce2ad9bb8074ada92744f6b2c040abb: Network error: Client connection negotiation failed: client connection to 127.11.102.1:33553: connect: Connection refused (error 111)
I20260620 06:07:21.558561 11672 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9HMp43/build/debug/bin/kudu with pid 19536
I20260620 06:07:21.565399 11672 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9HMp43/build/debug/bin/kudu with pid 19152
2026-06-20T06:07:21Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (3340 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781935603120653-11672-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 06:07:21 dist-test-slave-9hm5 krb5kdc[19798](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 06:07:21 dist-test-slave-9hm5 krb5kdc[19798](info): set up 2 sockets
Jun 20 06:07:21 dist-test-slave-9hm5 krb5kdc[19798](info): commencing operation
krb5kdc: starting...
W20260620 06:07:22.604349 11672 mini_kdc.cc:121] Time spent starting KDC: real 1.016s user 0.004s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781935603120653-11672-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781935603120653-11672-0/krb5kdc/test-user.keytab.
Jun 20 06:07:22 dist-test-slave-9hm5 krb5kdc[19798](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935642, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T06:07:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T06:07:22Z Disabled control of system clock
WARNING: no policy specified for kudu/127.11.102.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.102.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.102.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781935603120653-11672-0/krb5kdc/kudu_127.11.102.62.keytab.
Entry for principal kudu/127.11.102.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781935603120653-11672-0/krb5kdc/kudu_127.11.102.62.keytab.
I20260620 06:07:22.720523 11672 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9HMp43/build/debug/bin/kudu
/tmp/dist-test-task9HMp43/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781935603120653-11672-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781935603120653-11672-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781935603120653-11672-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781935603120653-11672-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.102.62:35235
--webserver_interface=127.11.102.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781935603120653-11672-0/krb5kdc/kudu_127.11.102.62.keytab
--principal=kudu/127.11.102.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.102.20:39171
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.102.62:35235
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781935603120653-11672-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781935603120653-11672-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781935603120653-11672-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 06:07:22.827030 19811 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:07:22.827282 19811 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:07:22.827325 19811 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:07:22.831298 19811 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 06:07:22.831359 19811 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:07:22.831382 19811 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:07:22.831453 19811 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 06:07:22.831491 19811 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 06:07:22.836649 19811 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.102.20:39171
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781935603120653-11672-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781935603120653-11672-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.102.62:35235
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781935603120653-11672-0/krb5kdc/kudu_127.11.102.62.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.102.62:35235
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781935603120653-11672-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.102.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19811
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781935603120653-11672-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 05:28:48 UTC on ccycloud.quasar-fixhms.root.comops.site
build id 4457020
I20260620 06:07:22.837642 19811 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:07:22.838645 19811 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 06:07:22.846870 19816 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:07:22.846894 19817 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:07:22.846993 19811 server_base.cc:1034] running on GCE node
W20260620 06:07:22.846891 19819 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:07:22.847707 19811 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:07:22.848809 19811 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:07:22.850001 19811 hybrid_clock.cc:648] HybridClock initialized: now 1781935642849979 us; error 30 us; skew 500 ppm
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-06-20T06:07:22Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1290 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781935603120653-11672-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 06:07:22 dist-test-slave-9hm5 krb5kdc[19824](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 06:07:22 dist-test-slave-9hm5 krb5kdc[19824](info): set up 2 sockets
Jun 20 06:07:22 dist-test-slave-9hm5 krb5kdc[19824](info): commencing operation
krb5kdc: starting...
W20260620 06:07:23.896437 11672 mini_kdc.cc:121] Time spent starting KDC: real 1.018s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781935603120653-11672-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781935603120653-11672-0/krb5kdc/test-user.keytab.
Jun 20 06:07:23 dist-test-slave-9hm5 krb5kdc[19824](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935643, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T06:07:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T06:07:23Z Disabled control of system clock
WARNING: no policy specified for kudu/127.11.102.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.102.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.102.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781935603120653-11672-0/krb5kdc/kudu_127.11.102.62.keytab.
Entry for principal kudu/127.11.102.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781935603120653-11672-0/krb5kdc/kudu_127.11.102.62.keytab.
I20260620 06:07:24.012540 11672 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9HMp43/build/debug/bin/kudu
/tmp/dist-test-task9HMp43/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781935603120653-11672-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781935603120653-11672-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781935603120653-11672-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781935603120653-11672-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.102.62:43493
--webserver_interface=127.11.102.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781935603120653-11672-0/krb5kdc/kudu_127.11.102.62.keytab
--principal=kudu/127.11.102.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.102.20:44309
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.102.62:43493
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781935603120653-11672-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781935603120653-11672-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781935603120653-11672-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 06:07:24.118523 19837 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:07:24.118773 19837 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:07:24.118814 19837 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:07:24.122728 19837 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 06:07:24.122781 19837 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:07:24.122800 19837 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:07:24.122817 19837 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 06:07:24.122833 19837 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 06:07:24.127934 19837 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.102.20:44309
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781935603120653-11672-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781935603120653-11672-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.102.62:43493
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781935603120653-11672-0/krb5kdc/kudu_127.11.102.62.keytab
--principal=kudu/127.11.102.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.102.62:43493
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781935603120653-11672-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.102.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19837
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781935603120653-11672-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 05:28:48 UTC on ccycloud.quasar-fixhms.root.comops.site
build id 4457020
I20260620 06:07:24.128896 19837 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:07:24.129837 19837 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 06:07:24.138223 19837 server_base.cc:1034] running on GCE node
W20260620 06:07:24.138164 19845 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:07:24.138131 19842 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:07:24.138145 19843 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:07:24.138849 19837 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:07:24.139945 19837 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:07:24.141093 19837 hybrid_clock.cc:648] HybridClock initialized: now 1781935644141066 us; error 53 us; skew 500 ppm
Jun 20 06:07:24 dist-test-slave-9hm5 krb5kdc[19824](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935644, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.102.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 06:07:24.148028 19837 init.cc:377] Logged in from keytab as kudu/127.11.102.62@KRBTEST.COM (short username kudu)
I20260620 06:07:24.148926 19837 webserver.cc:460] Webserver started at http://127.11.102.62:35255/ using document root <none> and password file <none>
I20260620 06:07:24.149470 19837 fs_manager.cc:359] Metadata directory not provided
I20260620 06:07:24.149561 19837 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:07:24.149783 19837 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 06:07:24.151827 19837 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781935603120653-11672-0/minicluster-data/master-0/data/instance:
uuid: "bf6b769c5e3544779150cd0d4a1fa643"
format_stamp: "Formatted at 2026-06-20 06:07:24 on dist-test-slave-9hm5"
server_key: "aeb268c0f538e358dbc993663541595e"
server_key_iv: "7b3264fa5f4978e1a8a02a48e2122d9a"
server_key_version: "encryptionkey@0"
I20260620 06:07:24.152372 19837 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781935603120653-11672-0/minicluster-data/master-0/wal/instance:
uuid: "bf6b769c5e3544779150cd0d4a1fa643"
format_stamp: "Formatted at 2026-06-20 06:07:24 on dist-test-slave-9hm5"
server_key: "aeb268c0f538e358dbc993663541595e"
server_key_iv: "7b3264fa5f4978e1a8a02a48e2122d9a"
server_key_version: "encryptionkey@0"
I20260620 06:07:24.155838 19837 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260620 06:07:24.158056 19852 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:07:24.159224 19837 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 06:07:24.159368 19837 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781935603120653-11672-0/minicluster-data/master-0/data,/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781935603120653-11672-0/minicluster-data/master-0/wal
uuid: "bf6b769c5e3544779150cd0d4a1fa643"
format_stamp: "Formatted at 2026-06-20 06:07:24 on dist-test-slave-9hm5"
server_key: "aeb268c0f538e358dbc993663541595e"
server_key_iv: "7b3264fa5f4978e1a8a02a48e2122d9a"
server_key_version: "encryptionkey@0"
I20260620 06:07:24.159471 19837 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781935603120653-11672-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781935603120653-11672-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781935603120653-11672-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:07:24.177699 19837 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:07:24.180486 19837 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:07:24.180686 19837 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:07:24.187564 19837 rpc_server.cc:307] RPC server started. Bound to: 127.11.102.62:43493
I20260620 06:07:24.187578 19904 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.102.62:43493 every 8 connection(s)
I20260620 06:07:24.188601 19837 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781935603120653-11672-0/minicluster-data/master-0/data/info.pb
I20260620 06:07:24.191280 19905 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 06:07:24.196563 19905 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bf6b769c5e3544779150cd0d4a1fa643: Bootstrap starting.
I20260620 06:07:24.198053 11672 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9HMp43/build/debug/bin/kudu as pid 19837
I20260620 06:07:24.198175 11672 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781935603120653-11672-0/minicluster-data/master-0/wal/instance
I20260620 06:07:24.198446 11672 external_mini_cluster.cc:1428] Setting key 849842eadf12c972f1e3b94c1f6b7374
I20260620 06:07:24.199363 19905 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bf6b769c5e3544779150cd0d4a1fa643: Neither blocks nor log segments found. Creating new log.
I20260620 06:07:24.200312 19905 log.cc:824] T 00000000000000000000000000000000 P bf6b769c5e3544779150cd0d4a1fa643: Log is configured to *not* fsync() on all Append() calls
I20260620 06:07:24.202508 19905 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bf6b769c5e3544779150cd0d4a1fa643: No bootstrap required, opened a new log
I20260620 06:07:24.206559 19905 raft_consensus.cc:348] T 00000000000000000000000000000000 P bf6b769c5e3544779150cd0d4a1fa643 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf6b769c5e3544779150cd0d4a1fa643" member_type: VOTER last_known_addr { host: "127.11.102.62" port: 43493 } }
I20260620 06:07:24.206777 19905 raft_consensus.cc:374] T 00000000000000000000000000000000 P bf6b769c5e3544779150cd0d4a1fa643 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 06:07:24.206836 19905 raft_consensus.cc:729] T 00000000000000000000000000000000 P bf6b769c5e3544779150cd0d4a1fa643 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bf6b769c5e3544779150cd0d4a1fa643, State: Initialized, Role: FOLLOWER
I20260620 06:07:24.207257 19905 consensus_queue.cc:260] T 00000000000000000000000000000000 P bf6b769c5e3544779150cd0d4a1fa643 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf6b769c5e3544779150cd0d4a1fa643" member_type: VOTER last_known_addr { host: "127.11.102.62" port: 43493 } }
I20260620 06:07:24.207412 19905 raft_consensus.cc:388] T 00000000000000000000000000000000 P bf6b769c5e3544779150cd0d4a1fa643 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 06:07:24.207505 19905 raft_consensus.cc:482] T 00000000000000000000000000000000 P bf6b769c5e3544779150cd0d4a1fa643 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
Jun 20 06:07:24 dist-test-slave-9hm5 krb5kdc[19824](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935643, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.102.62@KRBTEST.COM
I20260620 06:07:24.207654 19905 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bf6b769c5e3544779150cd0d4a1fa643 [term 0 FOLLOWER]: Advancing to term 1
I20260620 06:07:24.208665 19905 raft_consensus.cc:504] T 00000000000000000000000000000000 P bf6b769c5e3544779150cd0d4a1fa643 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf6b769c5e3544779150cd0d4a1fa643" member_type: VOTER last_known_addr { host: "127.11.102.62" port: 43493 } }
I20260620 06:07:24.209009 19905 leader_election.cc:302] T 00000000000000000000000000000000 P bf6b769c5e3544779150cd0d4a1fa643 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bf6b769c5e3544779150cd0d4a1fa643; no voters:
I20260620 06:07:24.209414 19905 leader_election.cc:288] T 00000000000000000000000000000000 P bf6b769c5e3544779150cd0d4a1fa643 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 06:07:24.209522 19910 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bf6b769c5e3544779150cd0d4a1fa643 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 06:07:24.209733 19910 raft_consensus.cc:686] T 00000000000000000000000000000000 P bf6b769c5e3544779150cd0d4a1fa643 [term 1 LEADER]: Becoming Leader. State: Replica: bf6b769c5e3544779150cd0d4a1fa643, State: Running, Role: LEADER
I20260620 06:07:24.210070 19910 consensus_queue.cc:237] T 00000000000000000000000000000000 P bf6b769c5e3544779150cd0d4a1fa643 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf6b769c5e3544779150cd0d4a1fa643" member_type: VOTER last_known_addr { host: "127.11.102.62" port: 43493 } }
I20260620 06:07:24.210723 19905 sys_catalog.cc:564] T 00000000000000000000000000000000 P bf6b769c5e3544779150cd0d4a1fa643 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 06:07:24.211817 19911 sys_catalog.cc:455] T 00000000000000000000000000000000 P bf6b769c5e3544779150cd0d4a1fa643 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bf6b769c5e3544779150cd0d4a1fa643" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf6b769c5e3544779150cd0d4a1fa643" member_type: VOTER last_known_addr { host: "127.11.102.62" port: 43493 } } }
I20260620 06:07:24.211931 19911 sys_catalog.cc:458] T 00000000000000000000000000000000 P bf6b769c5e3544779150cd0d4a1fa643 [sys.catalog]: This master's current role is: LEADER
I20260620 06:07:24.212312 19912 sys_catalog.cc:455] T 00000000000000000000000000000000 P bf6b769c5e3544779150cd0d4a1fa643 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bf6b769c5e3544779150cd0d4a1fa643. Latest consensus state: current_term: 1 leader_uuid: "bf6b769c5e3544779150cd0d4a1fa643" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf6b769c5e3544779150cd0d4a1fa643" member_type: VOTER last_known_addr { host: "127.11.102.62" port: 43493 } } }
I20260620 06:07:24.212481 19912 sys_catalog.cc:458] T 00000000000000000000000000000000 P bf6b769c5e3544779150cd0d4a1fa643 [sys.catalog]: This master's current role is: LEADER
I20260620 06:07:24.213920 19908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:07:24.199529 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39470 (local address 127.11.102.62:43493)
0620 06:07:24.199995 (+ 466us) server_negotiation.cc:206] Beginning negotiation
0620 06:07:24.200013 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0620 06:07:24.200056 (+ 43us) server_negotiation.cc:407] Connection header received
0620 06:07:24.200882 (+ 826us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:07:24.200901 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:07:24.201311 (+ 410us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:07:24.201667 (+ 356us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:07:24.202308 (+ 641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:07:24.203771 (+ 1463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:07:24.204615 (+ 844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:07:24.205406 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:07:24.205605 (+ 199us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:07:24.208207 (+ 2602us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:07:24.208212 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:07:24.208235 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:07:24.208261 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:07:24.210270 (+ 2009us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:07:24.210753 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:07:24.210759 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:07:24.210765 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:07:24.210907 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:07:24.211341 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:07:24.211346 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:07:24.211352 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:07:24.211928 (+ 576us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:07:24.212107 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0620 06:07:24.212613 (+ 506us) server_negotiation.cc:299] Negotiation successful
0620 06:07:24.212940 (+ 327us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":320,"thread_start_us":139,"threads_started":1}
W20260620 06:07:24.215571 19925 catalog_manager.cc:1558] T 00000000000000000000000000000000 P bf6b769c5e3544779150cd0d4a1fa643: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 06:07:24.215679 19925 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 06:07:24.215770 19919 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 06:07:24.216562 19919 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 06:07:24.221532 19919 catalog_manager.cc:1347] Generated new cluster ID: 8fd411ef1596432fb28b0e943ccf1367
I20260620 06:07:24.221588 19919 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 06:07:24.244627 19919 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 06:07:24.245620 19919 catalog_manager.cc:1504] Loading token signing keys...
I20260620 06:07:24.250398 19919 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bf6b769c5e3544779150cd0d4a1fa643: Generated new TSK 0
I20260620 06:07:24.251075 19919 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.11.102.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.102.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.102.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781935603120653-11672-0/krb5kdc/kudu_127.11.102.1.keytab.
Entry for principal kudu/127.11.102.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781935603120653-11672-0/krb5kdc/kudu_127.11.102.1.keytab.
I20260620 06:07:24.280375 11672 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9HMp43/build/debug/bin/kudu
/tmp/dist-test-task9HMp43/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781935603120653-11672-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781935603120653-11672-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781935603120653-11672-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781935603120653-11672-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.102.1:0
--local_ip_for_outbound_sockets=127.11.102.1
--webserver_interface=127.11.102.1
--webserver_port=0
--tserver_master_addrs=127.11.102.62:43493
--keytab_file=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781935603120653-11672-0/krb5kdc/kudu_127.11.102.1.keytab
--principal=kudu/127.11.102.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.102.20:44309
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781935603120653-11672-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781935603120653-11672-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781935603120653-11672-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 06:07:24.389288 19931 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:07:24.389550 19931 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:07:24.389593 19931 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:07:24.393572 19931 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:07:24.393625 19931 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:07:24.393682 19931 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.102.1
I20260620 06:07:24.398941 19931 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.102.20:44309
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781935603120653-11672-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781935603120653-11672-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781935603120653-11672-0/krb5kdc/kudu_127.11.102.1.keytab
--principal=kudu/127.11.102.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.102.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781935603120653-11672-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.11.102.1
--webserver_port=0
--tserver_master_addrs=127.11.102.62:43493
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19931
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.102.1
--log_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781935603120653-11672-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 05:28:48 UTC on ccycloud.quasar-fixhms.root.comops.site
build id 4457020
I20260620 06:07:24.399976 19931 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:07:24.400997 19931 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 06:07:24.410310 19939 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:07:24.410293 19937 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:07:24.410360 19931 server_base.cc:1034] running on GCE node
W20260620 06:07:24.410293 19936 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:07:24.411041 19931 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:07:24.411857 19931 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:07:24.413021 19931 hybrid_clock.cc:648] HybridClock initialized: now 1781935644412987 us; error 38 us; skew 500 ppm
Jun 20 06:07:24 dist-test-slave-9hm5 krb5kdc[19824](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935644, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.102.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 06:07:24.415783 19931 init.cc:377] Logged in from keytab as kudu/127.11.102.1@KRBTEST.COM (short username kudu)
I20260620 06:07:24.416633 19931 webserver.cc:460] Webserver started at http://127.11.102.1:42801/ using document root <none> and password file <none>
I20260620 06:07:24.417191 19931 fs_manager.cc:359] Metadata directory not provided
I20260620 06:07:24.417248 19931 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:07:24.417457 19931 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 06:07:24.419410 19931 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781935603120653-11672-0/minicluster-data/ts-0/data/instance:
uuid: "7e8ece56ae484a47a3b68c52742a5909"
format_stamp: "Formatted at 2026-06-20 06:07:24 on dist-test-slave-9hm5"
server_key: "9f274ff5a37573f333573f64a60e243f"
server_key_iv: "6d9807d8b995831599d5a51b0b937229"
server_key_version: "encryptionkey@0"
I20260620 06:07:24.420023 19931 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781935603120653-11672-0/minicluster-data/ts-0/wal/instance:
uuid: "7e8ece56ae484a47a3b68c52742a5909"
format_stamp: "Formatted at 2026-06-20 06:07:24 on dist-test-slave-9hm5"
server_key: "9f274ff5a37573f333573f64a60e243f"
server_key_iv: "6d9807d8b995831599d5a51b0b937229"
server_key_version: "encryptionkey@0"
I20260620 06:07:24.423520 19931 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260620 06:07:24.425941 19946 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:07:24.427145 19931 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 06:07:24.427261 19931 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781935603120653-11672-0/minicluster-data/ts-0/data,/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781935603120653-11672-0/minicluster-data/ts-0/wal
uuid: "7e8ece56ae484a47a3b68c52742a5909"
format_stamp: "Formatted at 2026-06-20 06:07:24 on dist-test-slave-9hm5"
server_key: "9f274ff5a37573f333573f64a60e243f"
server_key_iv: "6d9807d8b995831599d5a51b0b937229"
server_key_version: "encryptionkey@0"
I20260620 06:07:24.427364 19931 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781935603120653-11672-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781935603120653-11672-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781935603120653-11672-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:07:24.444990 19931 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:07:24.448050 19931 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:07:24.448257 19931 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:07:24.448863 19931 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 06:07:24.449687 19931 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 06:07:24.449733 19931 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 06:07:24.449803 19931 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 06:07:24.449836 19931 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 06:07:24.458554 19931 rpc_server.cc:307] RPC server started. Bound to: 127.11.102.1:37121
I20260620 06:07:24.458573 20059 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.102.1:37121 every 8 connection(s)
I20260620 06:07:24.459441 19931 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781935603120653-11672-0/minicluster-data/ts-0/data/info.pb
I20260620 06:07:24.466216 11672 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9HMp43/build/debug/bin/kudu as pid 19931
I20260620 06:07:24.466351 11672 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781935603120653-11672-0/minicluster-data/ts-0/wal/instance
I20260620 06:07:24.466647 11672 external_mini_cluster.cc:1428] Setting key b50d65df895f59d9197d154e8c240e15
Jun 20 06:07:24 dist-test-slave-9hm5 krb5kdc[19824](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935644, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.102.1@KRBTEST.COM for kudu/127.11.102.62@KRBTEST.COM
I20260620 06:07:24.473887 19908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:07:24.461337 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.102.1:47459 (local address 127.11.102.62:43493)
0620 06:07:24.461470 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0620 06:07:24.461473 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 06:07:24.462487 (+ 1014us) server_negotiation.cc:407] Connection header received
0620 06:07:24.463447 (+ 960us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:07:24.463450 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:07:24.463498 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:07:24.463573 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:07:24.464776 (+ 1203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:07:24.465308 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:07:24.466889 (+ 1581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:07:24.467168 (+ 279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:07:24.467314 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:07:24.470240 (+ 2926us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:07:24.470244 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:07:24.470247 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:07:24.470272 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:07:24.471890 (+ 1618us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:07:24.472404 (+ 514us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:07:24.472411 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:07:24.472414 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:07:24.472499 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:07:24.472833 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:07:24.472838 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:07:24.472840 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:07:24.473027 (+ 187us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:07:24.473130 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0620 06:07:24.473616 (+ 486us) server_negotiation.cc:299] Negotiation successful
0620 06:07:24.473732 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260620 06:07:24.474690 20062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:07:24.461664 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.102.62:43493 (local address 127.11.102.1:47459)
0620 06:07:24.462341 (+ 677us) negotiation.cc:107] Waiting for socket to connect
0620 06:07:24.462382 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0620 06:07:24.463267 (+ 885us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:07:24.463763 (+ 496us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:07:24.463770 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:07:24.464173 (+ 403us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:07:24.464645 (+ 472us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:07:24.464658 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:07:24.465454 (+ 796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:07:24.465457 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:07:24.466691 (+ 1234us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:07:24.466702 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:07:24.467369 (+ 667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:07:24.467384 (+ 15us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:07:24.467579 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:07:24.468318 (+ 739us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:07:24.468337 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:07:24.470076 (+ 1739us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:07:24.472015 (+ 1939us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:07:24.472021 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:07:24.472036 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:07:24.472300 (+ 264us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:07:24.472596 (+ 296us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:07:24.472599 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:07:24.472601 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:07:24.472739 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:07:24.473129 (+ 390us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:07:24.473136 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:07:24.473379 (+ 243us) client_negotiation.cc:769] Sending connection context
0620 06:07:24.473577 (+ 198us) client_negotiation.cc:240] Negotiation successful
0620 06:07:24.473885 (+ 308us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":498,"thread_start_us":107,"threads_started":1}
I20260620 06:07:24.476016 20060 heartbeater.cc:347] Connected to a master server at 127.11.102.62:43493
I20260620 06:07:24.476261 20060 heartbeater.cc:464] Registering TS with master...
I20260620 06:07:24.476734 20060 heartbeater.cc:511] Master 127.11.102.62:43493 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.11.102.2@KRBTEST.COM; defaulting to no policy
I20260620 06:07:24.478436 19869 ts_manager.cc:194] Registered new tserver with Master: 7e8ece56ae484a47a3b68c52742a5909 (127.11.102.1:37121)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.102.2@KRBTEST.COM" created.
I20260620 06:07:24.479919 19869 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.102.1@KRBTEST.COM'} at 127.11.102.1:47459
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.102.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781935603120653-11672-0/krb5kdc/kudu_127.11.102.2.keytab.
Entry for principal kudu/127.11.102.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781935603120653-11672-0/krb5kdc/kudu_127.11.102.2.keytab.
I20260620 06:07:24.494825 11672 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9HMp43/build/debug/bin/kudu
/tmp/dist-test-task9HMp43/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781935603120653-11672-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781935603120653-11672-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781935603120653-11672-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781935603120653-11672-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.102.2:0
--local_ip_for_outbound_sockets=127.11.102.2
--webserver_interface=127.11.102.2
--webserver_port=0
--tserver_master_addrs=127.11.102.62:43493
--keytab_file=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781935603120653-11672-0/krb5kdc/kudu_127.11.102.2.keytab
--principal=kudu/127.11.102.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.102.20:44309
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781935603120653-11672-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781935603120653-11672-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781935603120653-11672-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 06:07:24.601742 20065 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:07:24.602011 20065 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:07:24.602088 20065 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:07:24.606026 20065 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:07:24.606117 20065 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:07:24.606222 20065 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.102.2
I20260620 06:07:24.611354 20065 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.102.20:44309
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781935603120653-11672-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781935603120653-11672-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781935603120653-11672-0/krb5kdc/kudu_127.11.102.2.keytab
--principal=kudu/127.11.102.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.102.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781935603120653-11672-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.11.102.2
--webserver_port=0
--tserver_master_addrs=127.11.102.62:43493
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20065
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.102.2
--log_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781935603120653-11672-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 05:28:48 UTC on ccycloud.quasar-fixhms.root.comops.site
build id 4457020
I20260620 06:07:24.612413 20065 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:07:24.613381 20065 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 06:07:24.622723 20065 server_base.cc:1034] running on GCE node
W20260620 06:07:24.622701 20073 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:07:24.622712 20070 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:07:24.622859 20071 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:07:24.623265 20065 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:07:24.623910 20065 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:07:24.625094 20065 hybrid_clock.cc:648] HybridClock initialized: now 1781935644625086 us; error 30 us; skew 500 ppm
Jun 20 06:07:24 dist-test-slave-9hm5 krb5kdc[19824](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935644, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.102.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 06:07:24.627565 20065 init.cc:377] Logged in from keytab as kudu/127.11.102.2@KRBTEST.COM (short username kudu)
I20260620 06:07:24.628396 20065 webserver.cc:460] Webserver started at http://127.11.102.2:33581/ using document root <none> and password file <none>
I20260620 06:07:24.628919 20065 fs_manager.cc:359] Metadata directory not provided
I20260620 06:07:24.629004 20065 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:07:24.629233 20065 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 06:07:24.631170 20065 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781935603120653-11672-0/minicluster-data/ts-1/data/instance:
uuid: "3999cc84728945698e6aff45db976ceb"
format_stamp: "Formatted at 2026-06-20 06:07:24 on dist-test-slave-9hm5"
server_key: "422d3f5199d1490195ce0893ff800cab"
server_key_iv: "92f259815028bd938634209a0762a372"
server_key_version: "encryptionkey@0"
I20260620 06:07:24.631779 20065 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781935603120653-11672-0/minicluster-data/ts-1/wal/instance:
uuid: "3999cc84728945698e6aff45db976ceb"
format_stamp: "Formatted at 2026-06-20 06:07:24 on dist-test-slave-9hm5"
server_key: "422d3f5199d1490195ce0893ff800cab"
server_key_iv: "92f259815028bd938634209a0762a372"
server_key_version: "encryptionkey@0"
I20260620 06:07:24.635087 20065 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260620 06:07:24.637317 20080 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:07:24.638461 20065 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260620 06:07:24.638589 20065 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781935603120653-11672-0/minicluster-data/ts-1/data,/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781935603120653-11672-0/minicluster-data/ts-1/wal
uuid: "3999cc84728945698e6aff45db976ceb"
format_stamp: "Formatted at 2026-06-20 06:07:24 on dist-test-slave-9hm5"
server_key: "422d3f5199d1490195ce0893ff800cab"
server_key_iv: "92f259815028bd938634209a0762a372"
server_key_version: "encryptionkey@0"
I20260620 06:07:24.638691 20065 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781935603120653-11672-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781935603120653-11672-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781935603120653-11672-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:07:24.653745 20065 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:07:24.656694 20065 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:07:24.656946 20065 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:07:24.657612 20065 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 06:07:24.658501 20065 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 06:07:24.658587 20065 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 06:07:24.658672 20065 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 06:07:24.658730 20065 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 06:07:24.669128 20065 rpc_server.cc:307] RPC server started. Bound to: 127.11.102.2:41845
I20260620 06:07:24.669142 20193 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.102.2:41845 every 8 connection(s)
I20260620 06:07:24.670091 20065 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781935603120653-11672-0/minicluster-data/ts-1/data/info.pb
I20260620 06:07:24.670751 11672 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9HMp43/build/debug/bin/kudu as pid 20065
I20260620 06:07:24.670868 11672 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781935603120653-11672-0/minicluster-data/ts-1/wal/instance
I20260620 06:07:24.671192 11672 external_mini_cluster.cc:1428] Setting key 6807157bb3fb632bbfe422b9d5aa2681
WARNING: no policy specified for kudu/127.11.102.3@KRBTEST.COM; defaulting to no policy
Jun 20 06:07:24 dist-test-slave-9hm5 krb5kdc[19824](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935644, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.102.2@KRBTEST.COM for kudu/127.11.102.62@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.102.3@KRBTEST.COM" created.
I20260620 06:07:24.686290 19908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:07:24.672242 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.102.2:50373 (local address 127.11.102.62:43493)
0620 06:07:24.672421 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0620 06:07:24.672428 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 06:07:24.673853 (+ 1425us) server_negotiation.cc:407] Connection header received
0620 06:07:24.675077 (+ 1224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:07:24.675083 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:07:24.675143 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:07:24.675246 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:07:24.676799 (+ 1553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:07:24.677415 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:07:24.679291 (+ 1876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:07:24.679530 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:07:24.679697 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:07:24.682622 (+ 2925us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:07:24.682629 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:07:24.682633 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:07:24.682668 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:07:24.684197 (+ 1529us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:07:24.684722 (+ 525us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:07:24.684728 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:07:24.684730 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:07:24.684822 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:07:24.685171 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:07:24.685176 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:07:24.685178 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:07:24.685383 (+ 205us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:07:24.685489 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0620 06:07:24.686004 (+ 515us) server_negotiation.cc:299] Negotiation successful
0620 06:07:24.686125 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260620 06:07:24.687139 20196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:07:24.672859 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.102.62:43493 (local address 127.11.102.2:50373)
0620 06:07:24.673677 (+ 818us) negotiation.cc:107] Waiting for socket to connect
0620 06:07:24.673734 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0620 06:07:24.674868 (+ 1134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:07:24.675414 (+ 546us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:07:24.675420 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:07:24.675941 (+ 521us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:07:24.676635 (+ 694us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:07:24.676657 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:07:24.677552 (+ 895us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:07:24.677557 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:07:24.679158 (+ 1601us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:07:24.679173 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:07:24.679760 (+ 587us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:07:24.679764 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:07:24.680000 (+ 236us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:07:24.680655 (+ 655us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:07:24.680678 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:07:24.682411 (+ 1733us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:07:24.684311 (+ 1900us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:07:24.684317 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:07:24.684338 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:07:24.684609 (+ 271us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:07:24.684915 (+ 306us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:07:24.684919 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:07:24.684921 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:07:24.685074 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:07:24.685486 (+ 412us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:07:24.685494 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:07:24.685753 (+ 259us) client_negotiation.cc:769] Sending connection context
0620 06:07:24.685957 (+ 204us) client_negotiation.cc:240] Negotiation successful
0620 06:07:24.686298 (+ 341us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":569,"thread_start_us":127,"threads_started":1}
I20260620 06:07:24.688354 20194 heartbeater.cc:347] Connected to a master server at 127.11.102.62:43493
I20260620 06:07:24.688642 20194 heartbeater.cc:464] Registering TS with master...
I20260620 06:07:24.689299 20194 heartbeater.cc:511] Master 127.11.102.62:43493 requested a full tablet report, sending...
I20260620 06:07:24.690513 19869 ts_manager.cc:194] Registered new tserver with Master: 3999cc84728945698e6aff45db976ceb (127.11.102.2:41845)
I20260620 06:07:24.691342 19869 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.102.2@KRBTEST.COM'} at 127.11.102.2:50373
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.102.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781935603120653-11672-0/krb5kdc/kudu_127.11.102.3.keytab.
Entry for principal kudu/127.11.102.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781935603120653-11672-0/krb5kdc/kudu_127.11.102.3.keytab.
I20260620 06:07:24.700583 11672 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9HMp43/build/debug/bin/kudu
/tmp/dist-test-task9HMp43/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781935603120653-11672-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781935603120653-11672-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781935603120653-11672-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781935603120653-11672-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.102.3:0
--local_ip_for_outbound_sockets=127.11.102.3
--webserver_interface=127.11.102.3
--webserver_port=0
--tserver_master_addrs=127.11.102.62:43493
--keytab_file=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781935603120653-11672-0/krb5kdc/kudu_127.11.102.3.keytab
--principal=kudu/127.11.102.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.102.20:44309
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781935603120653-11672-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781935603120653-11672-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781935603120653-11672-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 06:07:24.807902 20199 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:07:24.808121 20199 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:07:24.808161 20199 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:07:24.812290 20199 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:07:24.812359 20199 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:07:24.812413 20199 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.102.3
I20260620 06:07:24.817348 20199 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.102.20:44309
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781935603120653-11672-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781935603120653-11672-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781935603120653-11672-0/krb5kdc/kudu_127.11.102.3.keytab
--principal=kudu/127.11.102.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.102.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781935603120653-11672-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.11.102.3
--webserver_port=0
--tserver_master_addrs=127.11.102.62:43493
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20199
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.102.3
--log_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781935603120653-11672-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 05:28:48 UTC on ccycloud.quasar-fixhms.root.comops.site
build id 4457020
I20260620 06:07:24.818176 20199 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:07:24.819057 20199 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 06:07:24.827859 20205 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:07:24.827893 20207 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:07:24.827859 20204 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:07:24.827919 20199 server_base.cc:1034] running on GCE node
I20260620 06:07:24.828428 20199 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:07:24.829030 20199 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:07:24.830221 20199 hybrid_clock.cc:648] HybridClock initialized: now 1781935644830190 us; error 46 us; skew 500 ppm
Jun 20 06:07:24 dist-test-slave-9hm5 krb5kdc[19824](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935644, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.102.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 06:07:24.833038 20199 init.cc:377] Logged in from keytab as kudu/127.11.102.3@KRBTEST.COM (short username kudu)
I20260620 06:07:24.833861 20199 webserver.cc:460] Webserver started at http://127.11.102.3:39965/ using document root <none> and password file <none>
I20260620 06:07:24.834381 20199 fs_manager.cc:359] Metadata directory not provided
I20260620 06:07:24.834467 20199 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:07:24.834683 20199 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 06:07:24.836637 20199 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781935603120653-11672-0/minicluster-data/ts-2/data/instance:
uuid: "db34226e3ab1496a89f52ed425ff8323"
format_stamp: "Formatted at 2026-06-20 06:07:24 on dist-test-slave-9hm5"
server_key: "d4e4606c6ed4acaa497eb398c137be14"
server_key_iv: "f10bf21fe1c9caab5b2aecf370d858fa"
server_key_version: "encryptionkey@0"
I20260620 06:07:24.837182 20199 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781935603120653-11672-0/minicluster-data/ts-2/wal/instance:
uuid: "db34226e3ab1496a89f52ed425ff8323"
format_stamp: "Formatted at 2026-06-20 06:07:24 on dist-test-slave-9hm5"
server_key: "d4e4606c6ed4acaa497eb398c137be14"
server_key_iv: "f10bf21fe1c9caab5b2aecf370d858fa"
server_key_version: "encryptionkey@0"
I20260620 06:07:24.840656 20199 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.004s
I20260620 06:07:24.843108 20214 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:07:24.844347 20199 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260620 06:07:24.844475 20199 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781935603120653-11672-0/minicluster-data/ts-2/data,/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781935603120653-11672-0/minicluster-data/ts-2/wal
uuid: "db34226e3ab1496a89f52ed425ff8323"
format_stamp: "Formatted at 2026-06-20 06:07:24 on dist-test-slave-9hm5"
server_key: "d4e4606c6ed4acaa497eb398c137be14"
server_key_iv: "f10bf21fe1c9caab5b2aecf370d858fa"
server_key_version: "encryptionkey@0"
I20260620 06:07:24.844581 20199 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781935603120653-11672-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781935603120653-11672-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781935603120653-11672-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:07:24.864050 20199 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:07:24.867148 20199 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:07:24.867340 20199 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:07:24.867949 20199 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 06:07:24.868702 20199 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 06:07:24.868779 20199 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 06:07:24.868845 20199 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 06:07:24.868896 20199 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 06:07:24.877465 20199 rpc_server.cc:307] RPC server started. Bound to: 127.11.102.3:34705
I20260620 06:07:24.877508 20327 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.102.3:34705 every 8 connection(s)
I20260620 06:07:24.878280 20199 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781935603120653-11672-0/minicluster-data/ts-2/data/info.pb
I20260620 06:07:24.886439 11672 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9HMp43/build/debug/bin/kudu as pid 20199
I20260620 06:07:24.886544 11672 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781935603120653-11672-0/minicluster-data/ts-2/wal/instance
Jun 20 06:07:24 dist-test-slave-9hm5 krb5kdc[19824](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935644, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.102.3@KRBTEST.COM for kudu/127.11.102.62@KRBTEST.COM
I20260620 06:07:24.886878 11672 external_mini_cluster.cc:1428] Setting key fece4a4644fe8680635499b2eb1d943e
I20260620 06:07:24.891260 19908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:07:24.879951 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.102.3:52467 (local address 127.11.102.62:43493)
0620 06:07:24.880075 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0620 06:07:24.880079 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 06:07:24.881056 (+ 977us) server_negotiation.cc:407] Connection header received
0620 06:07:24.881837 (+ 781us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:07:24.881840 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:07:24.881885 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:07:24.881958 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:07:24.883002 (+ 1044us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:07:24.883517 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:07:24.884582 (+ 1065us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:07:24.884793 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:07:24.884870 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:07:24.887373 (+ 2503us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:07:24.887377 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:07:24.887379 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:07:24.887406 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:07:24.889111 (+ 1705us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:07:24.889651 (+ 540us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:07:24.889657 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:07:24.889659 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:07:24.889753 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:07:24.890109 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:07:24.890115 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:07:24.890117 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:07:24.890312 (+ 195us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:07:24.890433 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0620 06:07:24.890930 (+ 497us) server_negotiation.cc:299] Negotiation successful
0620 06:07:24.891071 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260620 06:07:24.892164 20330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:07:24.880326 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.102.62:43493 (local address 127.11.102.3:52467)
0620 06:07:24.880917 (+ 591us) negotiation.cc:107] Waiting for socket to connect
0620 06:07:24.880953 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0620 06:07:24.881671 (+ 718us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:07:24.882100 (+ 429us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:07:24.882106 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:07:24.882443 (+ 337us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:07:24.882874 (+ 431us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:07:24.882886 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:07:24.883635 (+ 749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:07:24.883638 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:07:24.884482 (+ 844us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:07:24.884490 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:07:24.884915 (+ 425us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:07:24.884917 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:07:24.885056 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:07:24.885531 (+ 475us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:07:24.885546 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:07:24.887153 (+ 1607us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:07:24.889239 (+ 2086us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:07:24.889246 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:07:24.889261 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:07:24.889536 (+ 275us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:07:24.889853 (+ 317us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:07:24.889858 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:07:24.889860 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:07:24.890005 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:07:24.890431 (+ 426us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:07:24.890439 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:07:24.890682 (+ 243us) client_negotiation.cc:769] Sending connection context
0620 06:07:24.890885 (+ 203us) client_negotiation.cc:240] Negotiation successful
0620 06:07:24.891274 (+ 389us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":421,"thread_start_us":104,"threads_started":1}
I20260620 06:07:24.893255 20328 heartbeater.cc:347] Connected to a master server at 127.11.102.62:43493
I20260620 06:07:24.893498 20328 heartbeater.cc:464] Registering TS with master...
I20260620 06:07:24.893950 20328 heartbeater.cc:511] Master 127.11.102.62:43493 requested a full tablet report, sending...
I20260620 06:07:24.894950 19869 ts_manager.cc:194] Registered new tserver with Master: db34226e3ab1496a89f52ed425ff8323 (127.11.102.3:34705)
I20260620 06:07:24.895538 19869 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.102.3@KRBTEST.COM'} at 127.11.102.3:52467
I20260620 06:07:24.900591 11672 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.11.102.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.11.102.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.11.102.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781935603120653-11672-0/krb5kdc/oryx_127.11.102.1.keytab.
Entry for principal oryx/127.11.102.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781935603120653-11672-0/krb5kdc/oryx_127.11.102.1.keytab.
WARNING: no policy specified for oryx/127.11.102.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.11.102.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.11.102.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781935603120653-11672-0/krb5kdc/oryx_127.11.102.2.keytab.
Entry for principal oryx/127.11.102.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781935603120653-11672-0/krb5kdc/oryx_127.11.102.2.keytab.
WARNING: no policy specified for oryx/127.11.102.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.11.102.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.11.102.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781935603120653-11672-0/krb5kdc/oryx_127.11.102.3.keytab.
Entry for principal oryx/127.11.102.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781935603120653-11672-0/krb5kdc/oryx_127.11.102.3.keytab.
I20260620 06:07:24.974220 11672 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9HMp43/build/debug/bin/kudu with pid 19931
I20260620 06:07:24.979866 11672 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9HMp43/build/debug/bin/kudu with pid 20065
I20260620 06:07:24.985257 11672 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9HMp43/build/debug/bin/kudu with pid 20199
I20260620 06:07:24.991035 11672 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9HMp43/build/debug/bin/kudu with pid 19837
I20260620 06:07:24.997220 11672 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9HMp43/build/debug/bin/kudu
/tmp/dist-test-task9HMp43/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781935603120653-11672-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781935603120653-11672-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781935603120653-11672-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781935603120653-11672-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.102.62:43493
--webserver_interface=127.11.102.62
--webserver_port=35255
--keytab_file=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781935603120653-11672-0/krb5kdc/kudu_127.11.102.62.keytab
--principal=kudu/127.11.102.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.102.20:44309
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.102.62:43493
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781935603120653-11672-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781935603120653-11672-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781935603120653-11672-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 06:07:25.104573 20337 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:07:25.104839 20337 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:07:25.104913 20337 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:07:25.108894 20337 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 06:07:25.108950 20337 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:07:25.108969 20337 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:07:25.108986 20337 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 06:07:25.109036 20337 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 06:07:25.114215 20337 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.102.20:44309
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781935603120653-11672-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781935603120653-11672-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.102.62:43493
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781935603120653-11672-0/krb5kdc/kudu_127.11.102.62.keytab
--principal=kudu/127.11.102.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.102.62:43493
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781935603120653-11672-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.102.62
--webserver_port=35255
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20337
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781935603120653-11672-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 05:28:48 UTC on ccycloud.quasar-fixhms.root.comops.site
build id 4457020
I20260620 06:07:25.115254 20337 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:07:25.116341 20337 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 06:07:25.124549 20343 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:07:25.124593 20342 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:07:25.124567 20345 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:07:25.124960 20337 server_base.cc:1034] running on GCE node
I20260620 06:07:25.125442 20337 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:07:25.126458 20337 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:07:25.127665 20337 hybrid_clock.cc:648] HybridClock initialized: now 1781935645127585 us; error 45 us; skew 500 ppm
Jun 20 06:07:25 dist-test-slave-9hm5 krb5kdc[19824](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935645, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.102.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 06:07:25.130195 20337 init.cc:377] Logged in from keytab as kudu/127.11.102.62@KRBTEST.COM (short username kudu)
I20260620 06:07:25.130980 20337 webserver.cc:460] Webserver started at http://127.11.102.62:35255/ using document root <none> and password file <none>
I20260620 06:07:25.131479 20337 fs_manager.cc:359] Metadata directory not provided
I20260620 06:07:25.131546 20337 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:07:25.135223 20337 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260620 06:07:25.137384 20352 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:07:25.138514 20337 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260620 06:07:25.138636 20337 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781935603120653-11672-0/minicluster-data/master-0/data,/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781935603120653-11672-0/minicluster-data/master-0/wal
uuid: "bf6b769c5e3544779150cd0d4a1fa643"
format_stamp: "Formatted at 2026-06-20 06:07:24 on dist-test-slave-9hm5"
server_key: "aeb268c0f538e358dbc993663541595e"
server_key_iv: "7b3264fa5f4978e1a8a02a48e2122d9a"
server_key_version: "encryptionkey@0"
I20260620 06:07:25.139138 20337 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781935603120653-11672-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781935603120653-11672-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781935603120653-11672-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:07:25.174141 20337 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:07:25.177171 20337 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:07:25.177377 20337 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:07:25.184573 20337 rpc_server.cc:307] RPC server started. Bound to: 127.11.102.62:43493
I20260620 06:07:25.184588 20404 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.102.62:43493 every 8 connection(s)
I20260620 06:07:25.185629 20337 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781935603120653-11672-0/minicluster-data/master-0/data/info.pb
I20260620 06:07:25.189720 20405 sys_catalog.cc:263] Verifying existing consensus state
I20260620 06:07:25.191205 20405 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bf6b769c5e3544779150cd0d4a1fa643: Bootstrap starting.
I20260620 06:07:25.193361 11672 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9HMp43/build/debug/bin/kudu as pid 20337
I20260620 06:07:25.193809 11672 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9HMp43/build/debug/bin/kudu
/tmp/dist-test-task9HMp43/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781935603120653-11672-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781935603120653-11672-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781935603120653-11672-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781935603120653-11672-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.102.1:37121
--local_ip_for_outbound_sockets=127.11.102.1
--tserver_master_addrs=127.11.102.62:43493
--webserver_port=42801
--webserver_interface=127.11.102.1
--keytab_file=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781935603120653-11672-0/krb5kdc/kudu_127.11.102.1.keytab
--principal=kudu/127.11.102.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.102.20:44309
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.11.102.1
--keytab_file=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781935603120653-11672-0/krb5kdc/oryx_127.11.102.1.keytab with env {KRB5CCNAME=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781935603120653-11672-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781935603120653-11672-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781935603120653-11672-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 06:07:25.207497 20405 log.cc:824] T 00000000000000000000000000000000 P bf6b769c5e3544779150cd0d4a1fa643: Log is configured to *not* fsync() on all Append() calls
I20260620 06:07:25.213810 20405 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bf6b769c5e3544779150cd0d4a1fa643: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 06:07:25.214200 20405 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bf6b769c5e3544779150cd0d4a1fa643: Bootstrap complete.
I20260620 06:07:25.218403 20405 raft_consensus.cc:348] T 00000000000000000000000000000000 P bf6b769c5e3544779150cd0d4a1fa643 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf6b769c5e3544779150cd0d4a1fa643" member_type: VOTER last_known_addr { host: "127.11.102.62" port: 43493 } }
I20260620 06:07:25.218827 20405 raft_consensus.cc:729] T 00000000000000000000000000000000 P bf6b769c5e3544779150cd0d4a1fa643 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: bf6b769c5e3544779150cd0d4a1fa643, State: Initialized, Role: FOLLOWER
I20260620 06:07:25.219296 20405 consensus_queue.cc:260] T 00000000000000000000000000000000 P bf6b769c5e3544779150cd0d4a1fa643 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf6b769c5e3544779150cd0d4a1fa643" member_type: VOTER last_known_addr { host: "127.11.102.62" port: 43493 } }
I20260620 06:07:25.219439 20405 raft_consensus.cc:388] T 00000000000000000000000000000000 P bf6b769c5e3544779150cd0d4a1fa643 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 06:07:25.219527 20405 raft_consensus.cc:482] T 00000000000000000000000000000000 P bf6b769c5e3544779150cd0d4a1fa643 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 06:07:25.219667 20405 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bf6b769c5e3544779150cd0d4a1fa643 [term 1 FOLLOWER]: Advancing to term 2
I20260620 06:07:25.221489 20405 raft_consensus.cc:504] T 00000000000000000000000000000000 P bf6b769c5e3544779150cd0d4a1fa643 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf6b769c5e3544779150cd0d4a1fa643" member_type: VOTER last_known_addr { host: "127.11.102.62" port: 43493 } }
I20260620 06:07:25.221807 20405 leader_election.cc:302] T 00000000000000000000000000000000 P bf6b769c5e3544779150cd0d4a1fa643 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bf6b769c5e3544779150cd0d4a1fa643; no voters:
I20260620 06:07:25.222131 20405 leader_election.cc:288] T 00000000000000000000000000000000 P bf6b769c5e3544779150cd0d4a1fa643 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260620 06:07:25.222237 20409 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bf6b769c5e3544779150cd0d4a1fa643 [term 2 FOLLOWER]: Leader election won for term 2
I20260620 06:07:25.222436 20409 raft_consensus.cc:686] T 00000000000000000000000000000000 P bf6b769c5e3544779150cd0d4a1fa643 [term 2 LEADER]: Becoming Leader. State: Replica: bf6b769c5e3544779150cd0d4a1fa643, State: Running, Role: LEADER
I20260620 06:07:25.222770 20409 consensus_queue.cc:237] T 00000000000000000000000000000000 P bf6b769c5e3544779150cd0d4a1fa643 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf6b769c5e3544779150cd0d4a1fa643" member_type: VOTER last_known_addr { host: "127.11.102.62" port: 43493 } }
I20260620 06:07:25.223279 20405 sys_catalog.cc:564] T 00000000000000000000000000000000 P bf6b769c5e3544779150cd0d4a1fa643 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 06:07:25.224153 20410 sys_catalog.cc:455] T 00000000000000000000000000000000 P bf6b769c5e3544779150cd0d4a1fa643 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "bf6b769c5e3544779150cd0d4a1fa643" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf6b769c5e3544779150cd0d4a1fa643" member_type: VOTER last_known_addr { host: "127.11.102.62" port: 43493 } } }
I20260620 06:07:25.224277 20410 sys_catalog.cc:458] T 00000000000000000000000000000000 P bf6b769c5e3544779150cd0d4a1fa643 [sys.catalog]: This master's current role is: LEADER
I20260620 06:07:25.224545 20411 sys_catalog.cc:455] T 00000000000000000000000000000000 P bf6b769c5e3544779150cd0d4a1fa643 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bf6b769c5e3544779150cd0d4a1fa643. Latest consensus state: current_term: 2 leader_uuid: "bf6b769c5e3544779150cd0d4a1fa643" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf6b769c5e3544779150cd0d4a1fa643" member_type: VOTER last_known_addr { host: "127.11.102.62" port: 43493 } } }
I20260620 06:07:25.224615 20411 sys_catalog.cc:458] T 00000000000000000000000000000000 P bf6b769c5e3544779150cd0d4a1fa643 [sys.catalog]: This master's current role is: LEADER
I20260620 06:07:25.224983 20418 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 06:07:25.228195 20418 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 06:07:25.229117 20418 catalog_manager.cc:1259] Loaded cluster ID: 8fd411ef1596432fb28b0e943ccf1367
I20260620 06:07:25.229178 20418 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 06:07:25.231421 20418 catalog_manager.cc:1504] Loading token signing keys...
I20260620 06:07:25.232081 20418 catalog_manager.cc:5879] T 00000000000000000000000000000000 P bf6b769c5e3544779150cd0d4a1fa643: Loaded TSK: 0
I20260620 06:07:25.233273 20418 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260620 06:07:25.306890 20407 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:07:25.307127 20407 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:07:25.307170 20407 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:07:25.311162 20407 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:07:25.311219 20407 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:07:25.311275 20407 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.102.1
I20260620 06:07:25.316466 20407 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.102.20:44309
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781935603120653-11672-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781935603120653-11672-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781935603120653-11672-0/krb5kdc/oryx_127.11.102.1.keytab
--principal=oryx/127.11.102.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.102.1:37121
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781935603120653-11672-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.11.102.1
--webserver_port=42801
--tserver_master_addrs=127.11.102.62:43493
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20407
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.102.1
--log_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781935603120653-11672-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 05:28:48 UTC on ccycloud.quasar-fixhms.root.comops.site
build id 4457020
I20260620 06:07:25.317469 20407 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:07:25.318449 20407 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 06:07:25.327972 20432 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:07:25.328039 20431 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:07:25.327987 20434 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:07:25.328429 20407 server_base.cc:1034] running on GCE node
I20260620 06:07:25.328838 20407 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:07:25.329524 20407 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:07:25.330714 20407 hybrid_clock.cc:648] HybridClock initialized: now 1781935645330679 us; error 53 us; skew 500 ppm
Jun 20 06:07:25 dist-test-slave-9hm5 krb5kdc[19824](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935645, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.102.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 06:07:25.333576 20407 init.cc:377] Logged in from keytab as oryx/127.11.102.1@KRBTEST.COM (short username oryx)
I20260620 06:07:25.334441 20407 webserver.cc:460] Webserver started at http://127.11.102.1:42801/ using document root <none> and password file <none>
I20260620 06:07:25.335000 20407 fs_manager.cc:359] Metadata directory not provided
I20260620 06:07:25.335104 20407 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:07:25.339004 20407 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260620 06:07:25.341224 20441 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:07:25.342485 20407 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260620 06:07:25.342653 20407 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781935603120653-11672-0/minicluster-data/ts-0/data,/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781935603120653-11672-0/minicluster-data/ts-0/wal
uuid: "7e8ece56ae484a47a3b68c52742a5909"
format_stamp: "Formatted at 2026-06-20 06:07:24 on dist-test-slave-9hm5"
server_key: "9f274ff5a37573f333573f64a60e243f"
server_key_iv: "6d9807d8b995831599d5a51b0b937229"
server_key_version: "encryptionkey@0"
I20260620 06:07:25.343271 20407 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781935603120653-11672-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781935603120653-11672-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781935603120653-11672-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:07:25.367802 20407 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:07:25.370741 20407 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:07:25.370944 20407 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:07:25.371527 20407 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 06:07:25.372372 20407 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 06:07:25.372443 20407 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 06:07:25.372509 20407 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 06:07:25.372553 20407 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 06:07:25.381485 20407 rpc_server.cc:307] RPC server started. Bound to: 127.11.102.1:37121
I20260620 06:07:25.381510 20554 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.102.1:37121 every 8 connection(s)
I20260620 06:07:25.382472 20407 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781935603120653-11672-0/minicluster-data/ts-0/data/info.pb
I20260620 06:07:25.390340 11672 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9HMp43/build/debug/bin/kudu as pid 20407
I20260620 06:07:25.390838 11672 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9HMp43/build/debug/bin/kudu
/tmp/dist-test-task9HMp43/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781935603120653-11672-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781935603120653-11672-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781935603120653-11672-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781935603120653-11672-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.102.2:41845
--local_ip_for_outbound_sockets=127.11.102.2
--tserver_master_addrs=127.11.102.62:43493
--webserver_port=33581
--webserver_interface=127.11.102.2
--keytab_file=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781935603120653-11672-0/krb5kdc/kudu_127.11.102.2.keytab
--principal=kudu/127.11.102.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.102.20:44309
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.11.102.2
--keytab_file=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781935603120653-11672-0/krb5kdc/oryx_127.11.102.2.keytab with env {KRB5CCNAME=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781935603120653-11672-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781935603120653-11672-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781935603120653-11672-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 06:07:25 dist-test-slave-9hm5 krb5kdc[19824](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.102.1@KRBTEST.COM for oryx/127.11.102.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 06:07:25 dist-test-slave-9hm5 krb5kdc[19824](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.102.1@KRBTEST.COM for oryx/127.11.102.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 06:07:25.395857 20558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:07:25.384667 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.102.62:43493 (local address 127.11.102.1:49339)
0620 06:07:25.385366 (+ 699us) negotiation.cc:107] Waiting for socket to connect
0620 06:07:25.385405 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0620 06:07:25.386200 (+ 795us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:07:25.387287 (+ 1087us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:07:25.387294 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:07:25.387729 (+ 435us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:07:25.388195 (+ 466us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:07:25.388208 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:07:25.389461 (+ 1253us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:07:25.389466 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:07:25.390533 (+ 1067us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:07:25.390544 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:07:25.391408 (+ 864us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:07:25.391415 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:07:25.391671 (+ 256us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:07:25.392437 (+ 766us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:07:25.392454 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:07:25.395033 (+ 2579us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.102.62:43493: Server oryx/127.11.102.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":512,"thread_start_us":127,"threads_started":1}
W20260620 06:07:25.396340 20557 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 06:07:25.384570 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.102.1:49339 (local address 127.11.102.62:43493)
0620 06:07:25.385117 (+ 547us) server_negotiation.cc:206] Beginning negotiation
0620 06:07:25.385141 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0620 06:07:25.385523 (+ 382us) server_negotiation.cc:407] Connection header received
0620 06:07:25.386609 (+ 1086us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:07:25.386621 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:07:25.386903 (+ 282us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:07:25.387226 (+ 323us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:07:25.388363 (+ 1137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:07:25.389324 (+ 961us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:07:25.390661 (+ 1337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:07:25.391242 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:07:25.391367 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:07:25.395359 (+ 3992us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.102.1:49339: BlockingRecv error: recv got EOF from 127.11.102.1:49339 (error 108)
Metrics: {"server-negotiator.queue_time_us":362,"thread_start_us":131,"threads_started":1}
W20260620 06:07:25.396873 20555 heartbeater.cc:644] Failed to heartbeat to 127.11.102.62:43493 (0 consecutive failures): Not authorized: Failed to ping master at 127.11.102.62:43493: Client connection negotiation failed: client connection to 127.11.102.62:43493: Server oryx/127.11.102.62@KRBTEST.COM not found in Kerberos database
Jun 20 06:07:25 dist-test-slave-9hm5 krb5kdc[19824](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.102.1@KRBTEST.COM for oryx/127.11.102.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 06:07:25 dist-test-slave-9hm5 krb5kdc[19824](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.102.1@KRBTEST.COM for oryx/127.11.102.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 06:07:25.403352 20558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:07:25.397360 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.102.62:43493 (local address 127.11.102.1:45917)
0620 06:07:25.397530 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0620 06:07:25.397549 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 06:07:25.397652 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:07:25.397896 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:07:25.397899 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:07:25.398089 (+ 190us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:07:25.398231 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:07:25.398240 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:07:25.398948 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:07:25.398954 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:07:25.399981 (+ 1027us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:07:25.399995 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:07:25.400552 (+ 557us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:07:25.400555 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:07:25.400623 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:07:25.401170 (+ 547us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:07:25.401192 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:07:25.403211 (+ 2019us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.102.62:43493: Server oryx/127.11.102.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":62}
W20260620 06:07:25.403441 20557 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 06:07:25.397416 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.102.1:45917 (local address 127.11.102.62:43493)
0620 06:07:25.397525 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0620 06:07:25.397528 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 06:07:25.397615 (+ 87us) server_negotiation.cc:407] Connection header received
0620 06:07:25.397741 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:07:25.397743 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:07:25.397781 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:07:25.397846 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:07:25.398344 (+ 498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:07:25.398823 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:07:25.400099 (+ 1276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:07:25.400320 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:07:25.400438 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:07:25.403329 (+ 2891us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.102.1:45917: BlockingRecv error: recv got EOF from 127.11.102.1:45917 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
Jun 20 06:07:25 dist-test-slave-9hm5 krb5kdc[19824](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.102.1@KRBTEST.COM for oryx/127.11.102.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 06:07:25 dist-test-slave-9hm5 krb5kdc[19824](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.102.1@KRBTEST.COM for oryx/127.11.102.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 06:07:25.408820 20558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:07:25.403953 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.102.62:43493 (local address 127.11.102.1:49833)
0620 06:07:25.404059 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0620 06:07:25.404073 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 06:07:25.404145 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:07:25.404362 (+ 217us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:07:25.404365 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:07:25.404510 (+ 145us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:07:25.404583 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:07:25.404588 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:07:25.405381 (+ 793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:07:25.405386 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:07:25.405901 (+ 515us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:07:25.405908 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:07:25.406343 (+ 435us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:07:25.406346 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:07:25.406423 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:07:25.406899 (+ 476us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:07:25.406910 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:07:25.408686 (+ 1776us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.102.62:43493: Server oryx/127.11.102.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":45}
W20260620 06:07:25.408908 20557 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 06:07:25.404058 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.102.1:49833 (local address 127.11.102.62:43493)
0620 06:07:25.404159 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0620 06:07:25.404165 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 06:07:25.404182 (+ 17us) server_negotiation.cc:407] Connection header received
0620 06:07:25.404218 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:07:25.404221 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:07:25.404266 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:07:25.404348 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:07:25.404701 (+ 353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:07:25.405276 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:07:25.406015 (+ 739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:07:25.406236 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:07:25.406342 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:07:25.408793 (+ 2451us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.102.1:49833: BlockingRecv error: recv got EOF from 127.11.102.1:49833 (error 108)
Metrics: {"server-negotiator.queue_time_us":34}
W20260620 06:07:25.409024 20555 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260620 06:07:25.504050 20559 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:07:25.504302 20559 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:07:25.504343 20559 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:07:25.508350 20559 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:07:25.508402 20559 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:07:25.508497 20559 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.102.2
I20260620 06:07:25.513720 20559 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.102.20:44309
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781935603120653-11672-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781935603120653-11672-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781935603120653-11672-0/krb5kdc/oryx_127.11.102.2.keytab
--principal=oryx/127.11.102.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.102.2:41845
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781935603120653-11672-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.11.102.2
--webserver_port=33581
--tserver_master_addrs=127.11.102.62:43493
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20559
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.102.2
--log_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781935603120653-11672-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 05:28:48 UTC on ccycloud.quasar-fixhms.root.comops.site
build id 4457020
I20260620 06:07:25.514827 20559 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:07:25.515908 20559 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 06:07:25.525457 20564 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:07:25.525492 20565 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:07:25.525570 20559 server_base.cc:1034] running on GCE node
W20260620 06:07:25.525679 20567 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:07:25.526095 20559 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:07:25.526705 20559 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:07:25.527858 20559 hybrid_clock.cc:648] HybridClock initialized: now 1781935645527822 us; error 49 us; skew 500 ppm
Jun 20 06:07:25 dist-test-slave-9hm5 krb5kdc[19824](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935645, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.102.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 06:07:25.530480 20559 init.cc:377] Logged in from keytab as oryx/127.11.102.2@KRBTEST.COM (short username oryx)
I20260620 06:07:25.531373 20559 webserver.cc:460] Webserver started at http://127.11.102.2:33581/ using document root <none> and password file <none>
I20260620 06:07:25.531997 20559 fs_manager.cc:359] Metadata directory not provided
I20260620 06:07:25.532110 20559 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:07:25.535936 20559 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 06:07:25.537916 20574 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:07:25.539129 20559 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 06:07:25.539268 20559 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781935603120653-11672-0/minicluster-data/ts-1/data,/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781935603120653-11672-0/minicluster-data/ts-1/wal
uuid: "3999cc84728945698e6aff45db976ceb"
format_stamp: "Formatted at 2026-06-20 06:07:24 on dist-test-slave-9hm5"
server_key: "422d3f5199d1490195ce0893ff800cab"
server_key_iv: "92f259815028bd938634209a0762a372"
server_key_version: "encryptionkey@0"
I20260620 06:07:25.539847 20559 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781935603120653-11672-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781935603120653-11672-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781935603120653-11672-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:07:25.548518 20559 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:07:25.551241 20559 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:07:25.551435 20559 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:07:25.552064 20559 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 06:07:25.552845 20559 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 06:07:25.552917 20559 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 06:07:25.552981 20559 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 06:07:25.553033 20559 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 06:07:25.561755 20559 rpc_server.cc:307] RPC server started. Bound to: 127.11.102.2:41845
I20260620 06:07:25.561785 20687 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.102.2:41845 every 8 connection(s)
I20260620 06:07:25.562705 20559 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781935603120653-11672-0/minicluster-data/ts-1/data/info.pb
I20260620 06:07:25.567576 11672 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9HMp43/build/debug/bin/kudu as pid 20559
I20260620 06:07:25.568034 11672 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9HMp43/build/debug/bin/kudu
/tmp/dist-test-task9HMp43/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781935603120653-11672-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781935603120653-11672-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781935603120653-11672-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781935603120653-11672-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.102.3:34705
--local_ip_for_outbound_sockets=127.11.102.3
--tserver_master_addrs=127.11.102.62:43493
--webserver_port=39965
--webserver_interface=127.11.102.3
--keytab_file=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781935603120653-11672-0/krb5kdc/kudu_127.11.102.3.keytab
--principal=kudu/127.11.102.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.102.20:44309
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.11.102.3
--keytab_file=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781935603120653-11672-0/krb5kdc/oryx_127.11.102.3.keytab with env {KRB5CCNAME=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781935603120653-11672-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781935603120653-11672-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781935603120653-11672-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 06:07:25 dist-test-slave-9hm5 krb5kdc[19824](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.102.2@KRBTEST.COM for oryx/127.11.102.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 06:07:25 dist-test-slave-9hm5 krb5kdc[19824](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.102.2@KRBTEST.COM for oryx/127.11.102.62@KRBTEST.COM, Server not found in Kerberos database
W20260620 06:07:25.573698 20557 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 06:07:25.564503 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.102.2:32873 (local address 127.11.102.62:43493)
0620 06:07:25.564650 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0620 06:07:25.564654 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 06:07:25.565604 (+ 950us) server_negotiation.cc:407] Connection header received
0620 06:07:25.566486 (+ 882us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:07:25.566489 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:07:25.566537 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:07:25.566622 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:07:25.567840 (+ 1218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:07:25.568458 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:07:25.569579 (+ 1121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:07:25.569794 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:07:25.569902 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:07:25.573527 (+ 3625us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.102.2:32873: BlockingRecv error: recv got EOF from 127.11.102.2:32873 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
I20260620 06:07:25.574131 20690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:07:25.564877 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.102.62:43493 (local address 127.11.102.2:32873)
0620 06:07:25.565485 (+ 608us) negotiation.cc:107] Waiting for socket to connect
0620 06:07:25.565523 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0620 06:07:25.566273 (+ 750us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:07:25.566779 (+ 506us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:07:25.566785 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:07:25.567131 (+ 346us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:07:25.567671 (+ 540us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:07:25.567692 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:07:25.568581 (+ 889us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:07:25.568585 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:07:25.569449 (+ 864us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:07:25.569458 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:07:25.569948 (+ 490us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:07:25.569952 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:07:25.570116 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:07:25.570689 (+ 573us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:07:25.570713 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:07:25.573328 (+ 2615us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.102.62:43493: Server oryx/127.11.102.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":431,"thread_start_us":108,"threads_started":1}
W20260620 06:07:25.574723 20688 heartbeater.cc:644] Failed to heartbeat to 127.11.102.62:43493 (0 consecutive failures): Not authorized: Failed to ping master at 127.11.102.62:43493: Client connection negotiation failed: client connection to 127.11.102.62:43493: Server oryx/127.11.102.62@KRBTEST.COM not found in Kerberos database
Jun 20 06:07:25 dist-test-slave-9hm5 krb5kdc[19824](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.102.2@KRBTEST.COM for oryx/127.11.102.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 06:07:25 dist-test-slave-9hm5 krb5kdc[19824](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.102.2@KRBTEST.COM for oryx/127.11.102.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 06:07:25.580535 20690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:07:25.575152 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.102.62:43493 (local address 127.11.102.2:36665)
0620 06:07:25.575379 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0620 06:07:25.575392 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 06:07:25.575454 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:07:25.575710 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:07:25.575712 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:07:25.575837 (+ 125us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:07:25.575928 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:07:25.575933 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:07:25.576998 (+ 1065us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:07:25.577000 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:07:25.577518 (+ 518us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:07:25.577524 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:07:25.578134 (+ 610us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:07:25.578136 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:07:25.578200 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:07:25.578692 (+ 492us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:07:25.578703 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:07:25.580423 (+ 1720us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.102.62:43493: Server oryx/127.11.102.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":171}
W20260620 06:07:25.580647 20557 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 06:07:25.575268 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.102.2:36665 (local address 127.11.102.62:43493)
0620 06:07:25.575416 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0620 06:07:25.575423 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 06:07:25.575441 (+ 18us) server_negotiation.cc:407] Connection header received
0620 06:07:25.575530 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:07:25.575535 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:07:25.575591 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:07:25.575701 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:07:25.576040 (+ 339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:07:25.576894 (+ 854us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:07:25.577630 (+ 736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:07:25.578025 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:07:25.578149 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:07:25.580527 (+ 2378us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.102.2:36665: BlockingRecv error: recv got EOF from 127.11.102.2:36665 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
Jun 20 06:07:25 dist-test-slave-9hm5 krb5kdc[19824](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.102.2@KRBTEST.COM for oryx/127.11.102.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 06:07:25 dist-test-slave-9hm5 krb5kdc[19824](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.102.2@KRBTEST.COM for oryx/127.11.102.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 06:07:25.586302 20690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:07:25.581147 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.102.62:43493 (local address 127.11.102.2:59505)
0620 06:07:25.581316 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0620 06:07:25.581327 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 06:07:25.581379 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:07:25.581611 (+ 232us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:07:25.581613 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:07:25.581750 (+ 137us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:07:25.581814 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:07:25.581818 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:07:25.582795 (+ 977us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:07:25.582798 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:07:25.583298 (+ 500us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:07:25.583305 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:07:25.583985 (+ 680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:07:25.583989 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:07:25.584058 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:07:25.584513 (+ 455us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:07:25.584524 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:07:25.586224 (+ 1700us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.102.62:43493: Server oryx/127.11.102.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":105}
W20260620 06:07:25.586480 20688 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260620 06:07:25.586630 20557 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 06:07:25.581243 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.102.2:59505 (local address 127.11.102.62:43493)
0620 06:07:25.581350 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0620 06:07:25.581354 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 06:07:25.581367 (+ 13us) server_negotiation.cc:407] Connection header received
0620 06:07:25.581443 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:07:25.581446 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:07:25.581481 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:07:25.581556 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:07:25.581916 (+ 360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:07:25.582688 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:07:25.583402 (+ 714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:07:25.583771 (+ 369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:07:25.583868 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:07:25.586361 (+ 2493us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.102.2:59505: BlockingRecv error: recv got EOF from 127.11.102.2:59505 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
W20260620 06:07:25.684347 20691 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:07:25.684628 20691 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:07:25.684671 20691 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:07:25.688589 20691 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:07:25.688647 20691 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:07:25.688704 20691 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.102.3
I20260620 06:07:25.693842 20691 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.102.20:44309
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781935603120653-11672-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781935603120653-11672-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781935603120653-11672-0/krb5kdc/oryx_127.11.102.3.keytab
--principal=oryx/127.11.102.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.102.3:34705
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781935603120653-11672-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.11.102.3
--webserver_port=39965
--tserver_master_addrs=127.11.102.62:43493
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20691
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.102.3
--log_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781935603120653-11672-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 05:28:48 UTC on ccycloud.quasar-fixhms.root.comops.site
build id 4457020
I20260620 06:07:25.694841 20691 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:07:25.695832 20691 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 06:07:25.705523 20697 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:07:25.705565 20699 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:07:25.705643 20691 server_base.cc:1034] running on GCE node
W20260620 06:07:25.705813 20696 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:07:25.706221 20691 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:07:25.706908 20691 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:07:25.708088 20691 hybrid_clock.cc:648] HybridClock initialized: now 1781935645708058 us; error 51 us; skew 500 ppm
Jun 20 06:07:25 dist-test-slave-9hm5 krb5kdc[19824](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935645, etypes {rep=17 tkt=17 ses=17}, oryx/127.11.102.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 06:07:25.710678 20691 init.cc:377] Logged in from keytab as oryx/127.11.102.3@KRBTEST.COM (short username oryx)
I20260620 06:07:25.711501 20691 webserver.cc:460] Webserver started at http://127.11.102.3:39965/ using document root <none> and password file <none>
I20260620 06:07:25.712095 20691 fs_manager.cc:359] Metadata directory not provided
I20260620 06:07:25.712203 20691 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:07:25.715801 20691 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.004s sys 0.000s
I20260620 06:07:25.717718 20706 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:07:25.718879 20691 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 06:07:25.719005 20691 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781935603120653-11672-0/minicluster-data/ts-2/data,/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781935603120653-11672-0/minicluster-data/ts-2/wal
uuid: "db34226e3ab1496a89f52ed425ff8323"
format_stamp: "Formatted at 2026-06-20 06:07:24 on dist-test-slave-9hm5"
server_key: "d4e4606c6ed4acaa497eb398c137be14"
server_key_iv: "f10bf21fe1c9caab5b2aecf370d858fa"
server_key_version: "encryptionkey@0"
I20260620 06:07:25.719527 20691 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781935603120653-11672-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781935603120653-11672-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781935603120653-11672-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:07:25.748369 20691 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:07:25.751405 20691 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:07:25.751693 20691 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:07:25.752313 20691 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 06:07:25.753134 20691 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 06:07:25.753206 20691 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 06:07:25.753280 20691 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 06:07:25.753327 20691 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 06:07:25.762326 20691 rpc_server.cc:307] RPC server started. Bound to: 127.11.102.3:34705
I20260620 06:07:25.762370 20819 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.102.3:34705 every 8 connection(s)
I20260620 06:07:25.763281 20691 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781935603120653-11672-0/minicluster-data/ts-2/data/info.pb
I20260620 06:07:25.764770 11672 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9HMp43/build/debug/bin/kudu as pid 20691
Jun 20 06:07:25 dist-test-slave-9hm5 krb5kdc[19824](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.102.3@KRBTEST.COM for oryx/127.11.102.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 06:07:25 dist-test-slave-9hm5 krb5kdc[19824](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.102.3@KRBTEST.COM for oryx/127.11.102.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 06:07:25.774441 20824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:07:25.765764 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39486 (local address 127.11.102.62:43493)
0620 06:07:25.766001 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0620 06:07:25.766004 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 06:07:25.766014 (+ 10us) server_negotiation.cc:407] Connection header received
0620 06:07:25.766050 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:07:25.766053 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:07:25.766090 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:07:25.766190 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:07:25.766998 (+ 808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:07:25.767508 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:07:25.768431 (+ 923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:07:25.768657 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:07:25.768762 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:07:25.770396 (+ 1634us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:07:25.770401 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:07:25.770413 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:07:25.770437 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:07:25.772266 (+ 1829us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:07:25.772707 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:07:25.772714 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:07:25.772725 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:07:25.772879 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:07:25.773186 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:07:25.773189 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:07:25.773191 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:07:25.773612 (+ 421us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:07:25.773759 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0620 06:07:25.774057 (+ 298us) server_negotiation.cc:299] Negotiation successful
0620 06:07:25.774264 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":191,"thread_start_us":100,"threads_started":1}
W20260620 06:07:25.774586 20557 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 06:07:25.765091 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.102.3:53509 (local address 127.11.102.62:43493)
0620 06:07:25.765263 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0620 06:07:25.765270 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 06:07:25.766498 (+ 1228us) server_negotiation.cc:407] Connection header received
0620 06:07:25.767350 (+ 852us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:07:25.767353 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:07:25.767395 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:07:25.767465 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:07:25.768634 (+ 1169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:07:25.769193 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:07:25.770772 (+ 1579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:07:25.770982 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:07:25.771087 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:07:25.774459 (+ 3372us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.102.3:53509: BlockingRecv error: recv got EOF from 127.11.102.3:53509 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
I20260620 06:07:25.775252 20823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:07:25.765585 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.102.62:43493 (local address 127.11.102.3:53509)
0620 06:07:25.766340 (+ 755us) negotiation.cc:107] Waiting for socket to connect
0620 06:07:25.766391 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0620 06:07:25.767155 (+ 764us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:07:25.767696 (+ 541us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:07:25.767703 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:07:25.768031 (+ 328us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:07:25.768489 (+ 458us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:07:25.768502 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:07:25.769356 (+ 854us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:07:25.769359 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:07:25.770646 (+ 1287us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:07:25.770656 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:07:25.771125 (+ 469us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:07:25.771129 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:07:25.771300 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:07:25.771786 (+ 486us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:07:25.771802 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:07:25.774369 (+ 2567us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.102.62:43493: Server oryx/127.11.102.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":543,"thread_start_us":161,"threads_started":1}
W20260620 06:07:25.775748 20820 heartbeater.cc:644] Failed to heartbeat to 127.11.102.62:43493 (0 consecutive failures): Not authorized: Failed to ping master at 127.11.102.62:43493: Client connection negotiation failed: client connection to 127.11.102.62:43493: Server oryx/127.11.102.62@KRBTEST.COM not found in Kerberos database
Jun 20 06:07:25 dist-test-slave-9hm5 krb5kdc[19824](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.102.3@KRBTEST.COM for oryx/127.11.102.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 06:07:25 dist-test-slave-9hm5 krb5kdc[19824](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.102.3@KRBTEST.COM for oryx/127.11.102.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 06:07:25.780898 20823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:07:25.776166 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.102.62:43493 (local address 127.11.102.3:50241)
0620 06:07:25.776278 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0620 06:07:25.776293 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 06:07:25.776366 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:07:25.776631 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:07:25.776634 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:07:25.776792 (+ 158us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:07:25.776880 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:07:25.776885 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:07:25.777581 (+ 696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:07:25.777588 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:07:25.778168 (+ 580us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:07:25.778176 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:07:25.778627 (+ 451us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:07:25.778630 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:07:25.778718 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:07:25.779094 (+ 376us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:07:25.779104 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:07:25.780779 (+ 1675us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.102.62:43493: Server oryx/127.11.102.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":45}
W20260620 06:07:25.781033 20824 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 06:07:25.776208 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.102.3:50241 (local address 127.11.102.62:43493)
0620 06:07:25.776428 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0620 06:07:25.776432 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 06:07:25.776445 (+ 13us) server_negotiation.cc:407] Connection header received
0620 06:07:25.776478 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:07:25.776480 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:07:25.776518 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:07:25.776599 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:07:25.776991 (+ 392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:07:25.777478 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:07:25.778282 (+ 804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:07:25.778520 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:07:25.778627 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:07:25.780932 (+ 2305us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.102.3:50241: BlockingRecv error: recv got EOF from 127.11.102.3:50241 (error 108)
Metrics: {"server-negotiator.queue_time_us":156}
Jun 20 06:07:25 dist-test-slave-9hm5 krb5kdc[19824](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.102.3@KRBTEST.COM for oryx/127.11.102.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 06:07:25 dist-test-slave-9hm5 krb5kdc[19824](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.102.3@KRBTEST.COM for oryx/127.11.102.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 06:07:25.785996 20823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:07:25.781388 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.102.62:43493 (local address 127.11.102.3:48713)
0620 06:07:25.781506 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0620 06:07:25.781537 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0620 06:07:25.781611 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:07:25.781888 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:07:25.781890 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:07:25.782073 (+ 183us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:07:25.782141 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:07:25.782146 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:07:25.782838 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:07:25.782842 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:07:25.783351 (+ 509us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:07:25.783358 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:07:25.783864 (+ 506us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:07:25.783868 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:07:25.783952 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:07:25.784316 (+ 364us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:07:25.784326 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:07:25.785866 (+ 1540us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.102.62:43493: Server oryx/127.11.102.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":42}
W20260620 06:07:25.786079 20824 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 06:07:25.781474 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.102.3:48713 (local address 127.11.102.62:43493)
0620 06:07:25.781614 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0620 06:07:25.781618 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 06:07:25.781632 (+ 14us) server_negotiation.cc:407] Connection header received
0620 06:07:25.781699 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:07:25.781702 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:07:25.781750 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:07:25.781875 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:07:25.782253 (+ 378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:07:25.782714 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:07:25.783467 (+ 753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:07:25.783751 (+ 284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:07:25.783864 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:07:25.785970 (+ 2106us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.102.3:48713: BlockingRecv error: recv got EOF from 127.11.102.3:48713 (error 108)
Metrics: {"server-negotiator.queue_time_us":78}
W20260620 06:07:25.786203 20820 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Jun 20 06:07:26 dist-test-slave-9hm5 krb5kdc[19824](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.102.1@KRBTEST.COM for oryx/127.11.102.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 06:07:26 dist-test-slave-9hm5 krb5kdc[19824](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.102.1@KRBTEST.COM for oryx/127.11.102.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 06:07:26.415103 20825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:07:26.409576 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.102.62:43493 (local address 127.11.102.1:34211)
0620 06:07:26.409836 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0620 06:07:26.409854 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 06:07:26.409980 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:07:26.410304 (+ 324us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:07:26.410306 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:07:26.410534 (+ 228us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:07:26.410659 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:07:26.410666 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:07:26.411469 (+ 803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:07:26.411474 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:07:26.412057 (+ 583us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:07:26.412065 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:07:26.412515 (+ 450us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:07:26.412518 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:07:26.412601 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:07:26.413147 (+ 546us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:07:26.413161 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:07:26.414989 (+ 1828us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.102.62:43493: Server oryx/127.11.102.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":172,"thread_start_us":89,"threads_started":1}
W20260620 06:07:26.415275 20826 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 06:07:26.409652 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.102.1:34211 (local address 127.11.102.62:43493)
0620 06:07:26.409904 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0620 06:07:26.409910 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 06:07:26.409928 (+ 18us) server_negotiation.cc:407] Connection header received
0620 06:07:26.410094 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:07:26.410099 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:07:26.410167 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:07:26.410271 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:07:26.410765 (+ 494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:07:26.411357 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:07:26.412177 (+ 820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:07:26.412405 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:07:26.412510 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:07:26.415112 (+ 2602us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.102.1:34211: BlockingRecv error: recv got EOF from 127.11.102.1:34211 (error 108)
Metrics: {"server-negotiator.queue_time_us":176,"thread_start_us":98,"threads_started":1}
Jun 20 06:07:26 dist-test-slave-9hm5 krb5kdc[19824](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.102.2@KRBTEST.COM for oryx/127.11.102.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 06:07:26 dist-test-slave-9hm5 krb5kdc[19824](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.102.2@KRBTEST.COM for oryx/127.11.102.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 06:07:26.592638 20827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:07:26.587202 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.102.62:43493 (local address 127.11.102.2:38201)
0620 06:07:26.587474 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0620 06:07:26.587488 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 06:07:26.587575 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:07:26.587854 (+ 279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:07:26.587856 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:07:26.588025 (+ 169us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:07:26.588119 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:07:26.588124 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:07:26.588828 (+ 704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:07:26.588830 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:07:26.589380 (+ 550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:07:26.589386 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:07:26.589772 (+ 386us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:07:26.589774 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:07:26.589833 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:07:26.590402 (+ 569us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:07:26.590437 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:07:26.592464 (+ 2027us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.102.62:43493: Server oryx/127.11.102.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":165,"thread_start_us":85,"threads_started":1}
W20260620 06:07:26.592679 20826 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 06:07:26.587266 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.102.2:38201 (local address 127.11.102.62:43493)
0620 06:07:26.587511 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0620 06:07:26.587515 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 06:07:26.587532 (+ 17us) server_negotiation.cc:407] Connection header received
0620 06:07:26.587694 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:07:26.587697 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:07:26.587745 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:07:26.587817 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:07:26.588215 (+ 398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:07:26.588737 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:07:26.589473 (+ 736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:07:26.589678 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:07:26.589751 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:07:26.592578 (+ 2827us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.102.2:38201: BlockingRecv error: recv got EOF from 127.11.102.2:38201 (error 108)
Metrics: {"server-negotiator.queue_time_us":159}
Jun 20 06:07:26 dist-test-slave-9hm5 krb5kdc[19824](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.102.3@KRBTEST.COM for oryx/127.11.102.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 06:07:26 dist-test-slave-9hm5 krb5kdc[19824](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.102.3@KRBTEST.COM for oryx/127.11.102.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 06:07:26.792100 20828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:07:26.786899 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.102.62:43493 (local address 127.11.102.3:53833)
0620 06:07:26.787135 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0620 06:07:26.787149 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 06:07:26.787256 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:07:26.787478 (+ 222us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:07:26.787480 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:07:26.787672 (+ 192us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:07:26.787779 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:07:26.787784 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:07:26.788525 (+ 741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:07:26.788540 (+ 15us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:07:26.789109 (+ 569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:07:26.789116 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:07:26.789560 (+ 444us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:07:26.789563 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:07:26.789632 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:07:26.790085 (+ 453us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:07:26.790096 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:07:26.791941 (+ 1845us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.102.62:43493: Server oryx/127.11.102.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":78,"threads_started":1}
W20260620 06:07:26.792157 20826 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 06:07:26.786971 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.102.3:53833 (local address 127.11.102.62:43493)
0620 06:07:26.787113 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0620 06:07:26.787116 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 06:07:26.787234 (+ 118us) server_negotiation.cc:407] Connection header received
0620 06:07:26.787322 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:07:26.787324 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:07:26.787371 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:07:26.787466 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:07:26.787869 (+ 403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:07:26.788433 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:07:26.789232 (+ 799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:07:26.789457 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:07:26.789545 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:07:26.792052 (+ 2507us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.102.3:53833: BlockingRecv error: recv got EOF from 127.11.102.3:53833 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Jun 20 06:07:27 dist-test-slave-9hm5 krb5kdc[19824](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.102.1@KRBTEST.COM for oryx/127.11.102.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 06:07:27 dist-test-slave-9hm5 krb5kdc[19824](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.102.1@KRBTEST.COM for oryx/127.11.102.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 06:07:27.421319 20829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:07:27.415928 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.102.62:43493 (local address 127.11.102.1:59883)
0620 06:07:27.416237 (+ 309us) negotiation.cc:107] Waiting for socket to connect
0620 06:07:27.416250 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 06:07:27.416339 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:07:27.416621 (+ 282us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:07:27.416623 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:07:27.416794 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:07:27.416886 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:07:27.416891 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:07:27.417692 (+ 801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:07:27.417695 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:07:27.418250 (+ 555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:07:27.418258 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:07:27.418689 (+ 431us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:07:27.418691 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:07:27.418752 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:07:27.419257 (+ 505us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:07:27.419269 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:07:27.421185 (+ 1916us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.102.62:43493: Server oryx/127.11.102.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":226,"thread_start_us":78,"threads_started":1}
W20260620 06:07:27.421468 20830 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 06:07:27.415983 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.102.1:59883 (local address 127.11.102.62:43493)
0620 06:07:27.416276 (+ 293us) server_negotiation.cc:206] Beginning negotiation
0620 06:07:27.416279 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 06:07:27.416296 (+ 17us) server_negotiation.cc:407] Connection header received
0620 06:07:27.416455 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:07:27.416458 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:07:27.416504 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:07:27.416588 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:07:27.417000 (+ 412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:07:27.417567 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:07:27.418366 (+ 799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:07:27.418589 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:07:27.418667 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:07:27.421308 (+ 2641us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.102.1:59883: BlockingRecv error: recv got EOF from 127.11.102.1:59883 (error 108)
Metrics: {"server-negotiator.queue_time_us":220,"thread_start_us":123,"threads_started":1}
Jun 20 06:07:27 dist-test-slave-9hm5 krb5kdc[19824](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.102.2@KRBTEST.COM for oryx/127.11.102.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 06:07:27 dist-test-slave-9hm5 krb5kdc[19824](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.102.2@KRBTEST.COM for oryx/127.11.102.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 06:07:27.599283 20831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:07:27.593515 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.102.62:43493 (local address 127.11.102.2:47011)
0620 06:07:27.593773 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0620 06:07:27.593786 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 06:07:27.593882 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:07:27.594144 (+ 262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:07:27.594146 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:07:27.594350 (+ 204us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:07:27.594461 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:07:27.594467 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:07:27.595225 (+ 758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:07:27.595228 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:07:27.595842 (+ 614us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:07:27.595890 (+ 48us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:07:27.596437 (+ 547us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:07:27.596441 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:07:27.596551 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:07:27.597217 (+ 666us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:07:27.597229 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:07:27.599097 (+ 1868us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.102.62:43493: Server oryx/127.11.102.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":65,"threads_started":1}
W20260620 06:07:27.599304 20830 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 06:07:27.593622 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.102.2:47011 (local address 127.11.102.62:43493)
0620 06:07:27.593780 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0620 06:07:27.593784 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 06:07:27.593836 (+ 52us) server_negotiation.cc:407] Connection header received
0620 06:07:27.593963 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:07:27.593965 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:07:27.594006 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:07:27.594073 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:07:27.594581 (+ 508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:07:27.595117 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:07:27.596018 (+ 901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:07:27.596299 (+ 281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:07:27.596390 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:07:27.599211 (+ 2821us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.102.2:47011: BlockingRecv error: recv got EOF from 127.11.102.2:47011 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
Jun 20 06:07:27 dist-test-slave-9hm5 krb5kdc[19824](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.102.3@KRBTEST.COM for oryx/127.11.102.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 06:07:27 dist-test-slave-9hm5 krb5kdc[19824](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.102.3@KRBTEST.COM for oryx/127.11.102.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 06:07:27.802696 20832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:07:27.792969 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.102.62:43493 (local address 127.11.102.3:44017)
0620 06:07:27.795868 (+ 2899us) negotiation.cc:107] Waiting for socket to connect
0620 06:07:27.795890 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0620 06:07:27.796036 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:07:27.796355 (+ 319us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:07:27.796367 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:07:27.796622 (+ 255us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:07:27.796775 (+ 153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:07:27.796785 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:07:27.797982 (+ 1197us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:07:27.797987 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:07:27.799126 (+ 1139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:07:27.799140 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:07:27.799592 (+ 452us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:07:27.799597 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:07:27.799743 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:07:27.800295 (+ 552us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:07:27.800318 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:07:27.802524 (+ 2206us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.102.62:43493: Server oryx/127.11.102.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":2780,"thread_start_us":87,"threads_started":1}
W20260620 06:07:27.802742 20830 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 06:07:27.795692 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.102.3:44017 (local address 127.11.102.62:43493)
0620 06:07:27.795861 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0620 06:07:27.795866 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 06:07:27.795956 (+ 90us) server_negotiation.cc:407] Connection header received
0620 06:07:27.796150 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:07:27.796156 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:07:27.796220 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:07:27.796315 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:07:27.796918 (+ 603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:07:27.797846 (+ 928us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:07:27.799254 (+ 1408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:07:27.799473 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:07:27.799557 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:07:27.802640 (+ 3083us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.102.3:44017: BlockingRecv error: recv got EOF from 127.11.102.3:44017 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
Jun 20 06:07:28 dist-test-slave-9hm5 krb5kdc[19824](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.102.1@KRBTEST.COM for oryx/127.11.102.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 06:07:28 dist-test-slave-9hm5 krb5kdc[19824](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.102.1@KRBTEST.COM for oryx/127.11.102.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 06:07:28.427929 20833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:07:28.421997 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.102.62:43493 (local address 127.11.102.1:36023)
0620 06:07:28.422273 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0620 06:07:28.422287 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 06:07:28.422380 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:07:28.422629 (+ 249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:07:28.422631 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:07:28.422959 (+ 328us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:07:28.423078 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:07:28.423085 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:07:28.423895 (+ 810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:07:28.423901 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:07:28.424598 (+ 697us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:07:28.424607 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:07:28.425107 (+ 500us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:07:28.425109 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:07:28.425169 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:07:28.425793 (+ 624us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:07:28.425805 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:07:28.427814 (+ 2009us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.102.62:43493: Server oryx/127.11.102.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":98,"threads_started":1}
W20260620 06:07:28.428102 20834 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 06:07:28.422074 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.102.1:36023 (local address 127.11.102.62:43493)
0620 06:07:28.422316 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0620 06:07:28.422319 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 06:07:28.422333 (+ 14us) server_negotiation.cc:407] Connection header received
0620 06:07:28.422466 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:07:28.422468 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:07:28.422513 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:07:28.422590 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:07:28.423185 (+ 595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:07:28.423789 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:07:28.424785 (+ 996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:07:28.424989 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:07:28.425081 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:07:28.427930 (+ 2849us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.102.1:36023: BlockingRecv error: recv got EOF from 127.11.102.1:36023 (error 108)
Metrics: {"server-negotiator.queue_time_us":184,"thread_start_us":70,"threads_started":1}
Jun 20 06:07:28 dist-test-slave-9hm5 krb5kdc[19824](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.102.2@KRBTEST.COM for oryx/127.11.102.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 06:07:28 dist-test-slave-9hm5 krb5kdc[19824](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.102.2@KRBTEST.COM for oryx/127.11.102.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 06:07:28.605672 20835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:07:28.600075 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.102.62:43493 (local address 127.11.102.2:60573)
0620 06:07:28.600350 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0620 06:07:28.600363 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 06:07:28.600501 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:07:28.600772 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:07:28.600775 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:07:28.600984 (+ 209us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:07:28.601076 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:07:28.601082 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:07:28.601838 (+ 756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:07:28.601845 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:07:28.602399 (+ 554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:07:28.602406 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:07:28.602858 (+ 452us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:07:28.602861 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:07:28.602946 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:07:28.603504 (+ 558us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:07:28.603516 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:07:28.605561 (+ 2045us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.102.62:43493: Server oryx/127.11.102.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":196,"thread_start_us":67,"threads_started":1}
W20260620 06:07:28.605909 20834 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 06:07:28.600131 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.102.2:60573 (local address 127.11.102.62:43493)
0620 06:07:28.600285 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0620 06:07:28.600289 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 06:07:28.600441 (+ 152us) server_negotiation.cc:407] Connection header received
0620 06:07:28.600600 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:07:28.600604 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:07:28.600658 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:07:28.600744 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:07:28.601196 (+ 452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:07:28.601727 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:07:28.602530 (+ 803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:07:28.602749 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:07:28.602855 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:07:28.605696 (+ 2841us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.102.2:60573: BlockingRecv error: recv got EOF from 127.11.102.2:60573 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
Jun 20 06:07:28 dist-test-slave-9hm5 krb5kdc[19824](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.102.3@KRBTEST.COM for oryx/127.11.102.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 06:07:28 dist-test-slave-9hm5 krb5kdc[19824](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.102.3@KRBTEST.COM for oryx/127.11.102.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 06:07:28.808746 20836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:07:28.803505 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.102.62:43493 (local address 127.11.102.3:33971)
0620 06:07:28.803810 (+ 305us) negotiation.cc:107] Waiting for socket to connect
0620 06:07:28.803823 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 06:07:28.803920 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:07:28.804175 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:07:28.804177 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:07:28.804381 (+ 204us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:07:28.804477 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:07:28.804482 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:07:28.805191 (+ 709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:07:28.805193 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:07:28.805775 (+ 582us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:07:28.805782 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:07:28.806146 (+ 364us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:07:28.806148 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:07:28.806225 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:07:28.806806 (+ 581us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:07:28.806820 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:07:28.808627 (+ 1807us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.102.62:43493: Server oryx/127.11.102.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":211,"thread_start_us":107,"threads_started":1}
W20260620 06:07:28.808895 20834 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 06:07:28.803640 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.102.3:33971 (local address 127.11.102.62:43493)
0620 06:07:28.803769 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0620 06:07:28.803773 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 06:07:28.803896 (+ 123us) server_negotiation.cc:407] Connection header received
0620 06:07:28.803996 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:07:28.803999 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:07:28.804044 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:07:28.804111 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:07:28.804578 (+ 467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:07:28.805085 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:07:28.805870 (+ 785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:07:28.806061 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:07:28.806129 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:07:28.808745 (+ 2616us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.102.3:33971: BlockingRecv error: recv got EOF from 127.11.102.3:33971 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
Jun 20 06:07:29 dist-test-slave-9hm5 krb5kdc[19824](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.102.1@KRBTEST.COM for oryx/127.11.102.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 06:07:29 dist-test-slave-9hm5 krb5kdc[19824](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.102.1@KRBTEST.COM for oryx/127.11.102.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 06:07:29.436944 20837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:07:29.428720 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.102.62:43493 (local address 127.11.102.1:45231)
0620 06:07:29.428981 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0620 06:07:29.428997 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 06:07:29.429105 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:07:29.429588 (+ 483us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:07:29.429592 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:07:29.429765 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:07:29.429862 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:07:29.429868 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:07:29.430588 (+ 720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:07:29.430590 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:07:29.431152 (+ 562us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:07:29.431159 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:07:29.431578 (+ 419us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:07:29.431580 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:07:29.431720 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:07:29.434905 (+ 3185us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:07:29.434918 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:07:29.436833 (+ 1915us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.102.62:43493: Server oryx/127.11.102.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":100,"threads_started":1}
W20260620 06:07:29.437140 20838 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 06:07:29.429074 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.102.1:45231 (local address 127.11.102.62:43493)
0620 06:07:29.429303 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0620 06:07:29.429308 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 06:07:29.429335 (+ 27us) server_negotiation.cc:407] Connection header received
0620 06:07:29.429368 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:07:29.429371 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:07:29.429421 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:07:29.429530 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:07:29.429962 (+ 432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:07:29.430484 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:07:29.431268 (+ 784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:07:29.431475 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:07:29.431555 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:07:29.436962 (+ 5407us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.102.1:45231: BlockingRecv error: recv got EOF from 127.11.102.1:45231 (error 108)
Metrics: {"server-negotiator.queue_time_us":150,"thread_start_us":77,"threads_started":1}
Jun 20 06:07:29 dist-test-slave-9hm5 krb5kdc[19824](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.102.2@KRBTEST.COM for oryx/127.11.102.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 06:07:29 dist-test-slave-9hm5 krb5kdc[19824](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.102.2@KRBTEST.COM for oryx/127.11.102.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 06:07:29.611742 20839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:07:29.606432 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.102.62:43493 (local address 127.11.102.2:42461)
0620 06:07:29.606680 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0620 06:07:29.606694 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 06:07:29.606780 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:07:29.607080 (+ 300us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:07:29.607082 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:07:29.607253 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:07:29.607348 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:07:29.607353 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:07:29.608144 (+ 791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:07:29.608146 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:07:29.608686 (+ 540us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:07:29.608693 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:07:29.609153 (+ 460us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:07:29.609155 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:07:29.609214 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:07:29.609808 (+ 594us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:07:29.609820 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:07:29.611567 (+ 1747us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.102.62:43493: Server oryx/127.11.102.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":165,"thread_start_us":94,"threads_started":1}
W20260620 06:07:29.611846 20838 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 06:07:29.606579 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.102.2:42461 (local address 127.11.102.62:43493)
0620 06:07:29.606730 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0620 06:07:29.606734 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 06:07:29.606749 (+ 15us) server_negotiation.cc:407] Connection header received
0620 06:07:29.606868 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:07:29.606870 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:07:29.606915 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:07:29.607046 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:07:29.607455 (+ 409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:07:29.608050 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:07:29.608802 (+ 752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:07:29.609053 (+ 251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:07:29.609139 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:07:29.611713 (+ 2574us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.102.2:42461: BlockingRecv error: recv got EOF from 127.11.102.2:42461 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
Jun 20 06:07:29 dist-test-slave-9hm5 krb5kdc[19824](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.102.3@KRBTEST.COM for oryx/127.11.102.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 06:07:29 dist-test-slave-9hm5 krb5kdc[19824](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.102.3@KRBTEST.COM for oryx/127.11.102.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 06:07:29.815079 20840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:07:29.809493 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.102.62:43493 (local address 127.11.102.3:54897)
0620 06:07:29.809750 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0620 06:07:29.809763 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 06:07:29.809876 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:07:29.810210 (+ 334us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:07:29.810212 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:07:29.810402 (+ 190us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:07:29.810522 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:07:29.810527 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:07:29.811234 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:07:29.811236 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:07:29.811877 (+ 641us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:07:29.811888 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:07:29.812363 (+ 475us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:07:29.812367 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:07:29.812434 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:07:29.812970 (+ 536us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:07:29.812982 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:07:29.814966 (+ 1984us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.102.62:43493: Server oryx/127.11.102.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":80,"threads_started":1}
W20260620 06:07:29.815225 20838 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 06:07:29.809532 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.102.3:54897 (local address 127.11.102.62:43493)
0620 06:07:29.809650 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0620 06:07:29.809654 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 06:07:29.809813 (+ 159us) server_negotiation.cc:407] Connection header received
0620 06:07:29.810049 (+ 236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:07:29.810052 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:07:29.810128 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:07:29.810191 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:07:29.810617 (+ 426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:07:29.811141 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:07:29.812020 (+ 879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:07:29.812248 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:07:29.812324 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:07:29.815098 (+ 2774us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.102.3:54897: BlockingRecv error: recv got EOF from 127.11.102.3:54897 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
Jun 20 06:07:30 dist-test-slave-9hm5 krb5kdc[19824](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.102.1@KRBTEST.COM for oryx/127.11.102.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 06:07:30 dist-test-slave-9hm5 krb5kdc[19824](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.102.1@KRBTEST.COM for oryx/127.11.102.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 06:07:30.443578 20841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:07:30.437745 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.102.62:43493 (local address 127.11.102.1:54423)
0620 06:07:30.438053 (+ 308us) negotiation.cc:107] Waiting for socket to connect
0620 06:07:30.438067 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 06:07:30.438157 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:07:30.438439 (+ 282us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:07:30.438441 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:07:30.438677 (+ 236us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:07:30.438797 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:07:30.438805 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:07:30.439768 (+ 963us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:07:30.439772 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:07:30.440443 (+ 671us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:07:30.440452 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:07:30.440917 (+ 465us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:07:30.440920 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:07:30.440996 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:07:30.441451 (+ 455us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:07:30.441461 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:07:30.443466 (+ 2005us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.102.62:43493: Server oryx/127.11.102.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":225,"thread_start_us":86,"threads_started":1}
W20260620 06:07:30.443795 20842 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 06:07:30.437875 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.102.1:54423 (local address 127.11.102.62:43493)
0620 06:07:30.438101 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0620 06:07:30.438104 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 06:07:30.438118 (+ 14us) server_negotiation.cc:407] Connection header received
0620 06:07:30.438247 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:07:30.438250 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:07:30.438289 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:07:30.438370 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:07:30.438954 (+ 584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:07:30.439665 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:07:30.440552 (+ 887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:07:30.440795 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:07:30.440883 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:07:30.443604 (+ 2721us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.102.1:54423: BlockingRecv error: recv got EOF from 127.11.102.1:54423 (error 108)
Metrics: {"server-negotiator.queue_time_us":162,"thread_start_us":52,"threads_started":1}
Jun 20 06:07:30 dist-test-slave-9hm5 krb5kdc[19824](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.102.2@KRBTEST.COM for oryx/127.11.102.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 06:07:30 dist-test-slave-9hm5 krb5kdc[19824](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.102.2@KRBTEST.COM for oryx/127.11.102.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 06:07:30.617944 20843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:07:30.612540 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.102.62:43493 (local address 127.11.102.2:47397)
0620 06:07:30.612860 (+ 320us) negotiation.cc:107] Waiting for socket to connect
0620 06:07:30.612878 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 06:07:30.612997 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:07:30.613309 (+ 312us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:07:30.613315 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:07:30.613589 (+ 274us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:07:30.613701 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:07:30.613708 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:07:30.614446 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:07:30.614449 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:07:30.615033 (+ 584us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:07:30.615040 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:07:30.615445 (+ 405us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:07:30.615448 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:07:30.615517 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:07:30.616050 (+ 533us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:07:30.616061 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:07:30.617814 (+ 1753us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.102.62:43493: Server oryx/127.11.102.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":238,"thread_start_us":89,"threads_started":1}
W20260620 06:07:30.618037 20842 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 06:07:30.612679 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.102.2:47397 (local address 127.11.102.62:43493)
0620 06:07:30.612801 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0620 06:07:30.612806 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 06:07:30.612957 (+ 151us) server_negotiation.cc:407] Connection header received
0620 06:07:30.613112 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:07:30.613118 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:07:30.613181 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:07:30.613261 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:07:30.613811 (+ 550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:07:30.614330 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:07:30.615136 (+ 806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:07:30.615341 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:07:30.615415 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:07:30.617937 (+ 2522us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.102.2:47397: BlockingRecv error: recv got EOF from 127.11.102.2:47397 (error 108)
Metrics: {"server-negotiator.queue_time_us":30}
Jun 20 06:07:30 dist-test-slave-9hm5 krb5kdc[19824](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.102.3@KRBTEST.COM for oryx/127.11.102.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 06:07:30 dist-test-slave-9hm5 krb5kdc[19824](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.102.3@KRBTEST.COM for oryx/127.11.102.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 06:07:30.821527 20844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:07:30.815932 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.102.62:43493 (local address 127.11.102.3:43155)
0620 06:07:30.816271 (+ 339us) negotiation.cc:107] Waiting for socket to connect
0620 06:07:30.816293 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0620 06:07:30.816420 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:07:30.816714 (+ 294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:07:30.816717 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:07:30.816946 (+ 229us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:07:30.817038 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:07:30.817044 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:07:30.817779 (+ 735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:07:30.817785 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:07:30.818404 (+ 619us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:07:30.818418 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:07:30.818991 (+ 573us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:07:30.818995 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:07:30.819087 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:07:30.819706 (+ 619us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:07:30.819722 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:07:30.821410 (+ 1688us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.102.62:43493: Server oryx/127.11.102.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":258,"thread_start_us":84,"threads_started":1}
W20260620 06:07:30.821687 20842 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 06:07:30.816045 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.102.3:43155 (local address 127.11.102.62:43493)
0620 06:07:30.816223 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0620 06:07:30.816228 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 06:07:30.816370 (+ 142us) server_negotiation.cc:407] Connection header received
0620 06:07:30.816560 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:07:30.816563 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:07:30.816609 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:07:30.816692 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:07:30.817161 (+ 469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:07:30.817673 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:07:30.818568 (+ 895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:07:30.818897 (+ 329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:07:30.818973 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:07:30.821537 (+ 2564us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.102.3:43155: BlockingRecv error: recv got EOF from 127.11.102.3:43155 (error 108)
Metrics: {"server-negotiator.queue_time_us":83}
Jun 20 06:07:31 dist-test-slave-9hm5 krb5kdc[19824](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.102.1@KRBTEST.COM for oryx/127.11.102.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 06:07:31 dist-test-slave-9hm5 krb5kdc[19824](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.102.1@KRBTEST.COM for oryx/127.11.102.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 06:07:31.450172 20845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:07:31.444456 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.102.62:43493 (local address 127.11.102.1:57351)
0620 06:07:31.444723 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0620 06:07:31.444736 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 06:07:31.444832 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:07:31.445085 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:07:31.445088 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:07:31.445258 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:07:31.445351 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:07:31.445355 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:07:31.446131 (+ 776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:07:31.446133 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:07:31.446861 (+ 728us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:07:31.446870 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:07:31.447291 (+ 421us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:07:31.447294 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:07:31.447358 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:07:31.447940 (+ 582us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:07:31.447961 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:07:31.450009 (+ 2048us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.102.62:43493: Server oryx/127.11.102.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":100,"threads_started":1}
W20260620 06:07:31.450253 20846 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 06:07:31.444487 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.102.1:57351 (local address 127.11.102.62:43493)
0620 06:07:31.444717 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0620 06:07:31.444720 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 06:07:31.444802 (+ 82us) server_negotiation.cc:407] Connection header received
0620 06:07:31.444906 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:07:31.444908 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:07:31.444951 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:07:31.445033 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:07:31.445465 (+ 432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:07:31.446003 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:07:31.446958 (+ 955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:07:31.447184 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:07:31.447265 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:07:31.450129 (+ 2864us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.102.1:57351: BlockingRecv error: recv got EOF from 127.11.102.1:57351 (error 108)
Metrics: {"server-negotiator.queue_time_us":156,"thread_start_us":92,"threads_started":1}
Jun 20 06:07:31 dist-test-slave-9hm5 krb5kdc[19824](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.102.2@KRBTEST.COM for oryx/127.11.102.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 06:07:31 dist-test-slave-9hm5 krb5kdc[19824](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.102.2@KRBTEST.COM for oryx/127.11.102.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 06:07:31.624125 20847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:07:31.618686 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.102.62:43493 (local address 127.11.102.2:49027)
0620 06:07:31.618951 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0620 06:07:31.618964 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 06:07:31.619069 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:07:31.619294 (+ 225us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:07:31.619296 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:07:31.619478 (+ 182us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:07:31.619573 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:07:31.619578 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:07:31.620401 (+ 823us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:07:31.620405 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:07:31.621028 (+ 623us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:07:31.621038 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:07:31.621533 (+ 495us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:07:31.621538 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:07:31.621645 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:07:31.622098 (+ 453us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:07:31.622109 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:07:31.624009 (+ 1900us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.102.62:43493: Server oryx/127.11.102.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":197,"thread_start_us":108,"threads_started":1}
W20260620 06:07:31.624286 20846 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 06:07:31.618770 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.102.2:49027 (local address 127.11.102.62:43493)
0620 06:07:31.618930 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0620 06:07:31.618935 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 06:07:31.619036 (+ 101us) server_negotiation.cc:407] Connection header received
0620 06:07:31.619149 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:07:31.619151 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:07:31.619196 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:07:31.619262 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:07:31.619741 (+ 479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:07:31.620264 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:07:31.621133 (+ 869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:07:31.621404 (+ 271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:07:31.621492 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:07:31.624141 (+ 2649us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.102.2:49027: BlockingRecv error: recv got EOF from 127.11.102.2:49027 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
Jun 20 06:07:31 dist-test-slave-9hm5 krb5kdc[19824](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.102.3@KRBTEST.COM for oryx/127.11.102.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 06:07:31 dist-test-slave-9hm5 krb5kdc[19824](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.102.3@KRBTEST.COM for oryx/127.11.102.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 06:07:31.828150 20848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:07:31.822226 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.102.62:43493 (local address 127.11.102.3:43035)
0620 06:07:31.822480 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0620 06:07:31.822492 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 06:07:31.822606 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:07:31.822921 (+ 315us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:07:31.822923 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:07:31.823140 (+ 217us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:07:31.823259 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:07:31.823265 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:07:31.824161 (+ 896us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:07:31.824164 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:07:31.824755 (+ 591us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:07:31.824765 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:07:31.825321 (+ 556us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:07:31.825325 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:07:31.825435 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:07:31.825904 (+ 469us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:07:31.825915 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:07:31.827992 (+ 2077us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.102.62:43493: Server oryx/127.11.102.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":173,"thread_start_us":98,"threads_started":1}
W20260620 06:07:31.828233 20846 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 06:07:31.822329 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.102.3:43035 (local address 127.11.102.62:43493)
0620 06:07:31.822458 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0620 06:07:31.822461 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 06:07:31.822577 (+ 116us) server_negotiation.cc:407] Connection header received
0620 06:07:31.822694 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:07:31.822696 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:07:31.822743 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:07:31.822904 (+ 161us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:07:31.823391 (+ 487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:07:31.824039 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:07:31.824902 (+ 863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:07:31.825179 (+ 277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:07:31.825306 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:07:31.828106 (+ 2800us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.102.3:43035: BlockingRecv error: recv got EOF from 127.11.102.3:43035 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
Jun 20 06:07:32 dist-test-slave-9hm5 krb5kdc[19824](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.102.1@KRBTEST.COM for oryx/127.11.102.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 06:07:32 dist-test-slave-9hm5 krb5kdc[19824](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.102.1@KRBTEST.COM for oryx/127.11.102.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 06:07:32.456686 20850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:07:32.450927 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.102.62:43493 (local address 127.11.102.1:40983)
0620 06:07:32.451272 (+ 345us) negotiation.cc:107] Waiting for socket to connect
0620 06:07:32.451293 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0620 06:07:32.451426 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:07:32.451732 (+ 306us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:07:32.451734 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:07:32.451924 (+ 190us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:07:32.452040 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:07:32.452046 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:07:32.452783 (+ 737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:07:32.452785 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:07:32.453367 (+ 582us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:07:32.453375 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:07:32.453854 (+ 479us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:07:32.453857 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:07:32.453924 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:07:32.454468 (+ 544us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:07:32.454481 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:07:32.456522 (+ 2041us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.102.62:43493: Server oryx/127.11.102.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":262,"thread_start_us":74,"threads_started":1}
W20260620 06:07:32.456862 20849 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 06:07:32.450974 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.102.1:40983 (local address 127.11.102.62:43493)
0620 06:07:32.451233 (+ 259us) server_negotiation.cc:206] Beginning negotiation
0620 06:07:32.451238 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 06:07:32.451369 (+ 131us) server_negotiation.cc:407] Connection header received
0620 06:07:32.451522 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:07:32.451525 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:07:32.451575 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:07:32.451704 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:07:32.452147 (+ 443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:07:32.452675 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:07:32.453496 (+ 821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:07:32.453741 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:07:32.453831 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:07:32.456691 (+ 2860us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.102.1:40983: BlockingRecv error: recv got EOF from 127.11.102.1:40983 (error 108)
Metrics: {"server-negotiator.queue_time_us":171,"thread_start_us":89,"threads_started":1}
Jun 20 06:07:32 dist-test-slave-9hm5 krb5kdc[19824](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.102.2@KRBTEST.COM for oryx/127.11.102.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 06:07:32 dist-test-slave-9hm5 krb5kdc[19824](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.102.2@KRBTEST.COM for oryx/127.11.102.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 06:07:32.630103 20851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:07:32.624923 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.102.62:43493 (local address 127.11.102.2:44723)
0620 06:07:32.625239 (+ 316us) negotiation.cc:107] Waiting for socket to connect
0620 06:07:32.625252 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 06:07:32.625344 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:07:32.625546 (+ 202us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:07:32.625548 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:07:32.625725 (+ 177us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:07:32.625815 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:07:32.625820 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:07:32.626509 (+ 689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:07:32.626511 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:07:32.627062 (+ 551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:07:32.627070 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:07:32.627574 (+ 504us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:07:32.627577 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:07:32.627702 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:07:32.628279 (+ 577us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:07:32.628290 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:07:32.629993 (+ 1703us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.102.62:43493: Server oryx/127.11.102.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":237,"thread_start_us":96,"threads_started":1}
W20260620 06:07:32.630254 20849 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 06:07:32.624921 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.102.2:44723 (local address 127.11.102.62:43493)
0620 06:07:32.625077 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0620 06:07:32.625080 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 06:07:32.625314 (+ 234us) server_negotiation.cc:407] Connection header received
0620 06:07:32.625407 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:07:32.625410 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:07:32.625450 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:07:32.625515 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:07:32.625904 (+ 389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:07:32.626418 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:07:32.627191 (+ 773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:07:32.627457 (+ 266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:07:32.627561 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:07:32.630115 (+ 2554us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.102.2:44723: BlockingRecv error: recv got EOF from 127.11.102.2:44723 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
Jun 20 06:07:32 dist-test-slave-9hm5 krb5kdc[19824](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.102.3@KRBTEST.COM for oryx/127.11.102.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 06:07:32 dist-test-slave-9hm5 krb5kdc[19824](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.102.3@KRBTEST.COM for oryx/127.11.102.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 06:07:32.834483 20852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:07:32.828883 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.102.62:43493 (local address 127.11.102.3:46029)
0620 06:07:32.829188 (+ 305us) negotiation.cc:107] Waiting for socket to connect
0620 06:07:32.829200 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 06:07:32.829328 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:07:32.829585 (+ 257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:07:32.829588 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:07:32.829793 (+ 205us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:07:32.829879 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:07:32.829885 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:07:32.830650 (+ 765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:07:32.830655 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:07:32.831333 (+ 678us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:07:32.831347 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:07:32.831828 (+ 481us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:07:32.831832 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:07:32.831926 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:07:32.832454 (+ 528us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:07:32.832468 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:07:32.834309 (+ 1841us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.102.62:43493: Server oryx/127.11.102.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":237,"thread_start_us":76,"threads_started":1}
W20260620 06:07:32.834525 20849 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 06:07:32.828979 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.102.3:46029 (local address 127.11.102.62:43493)
0620 06:07:32.829107 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0620 06:07:32.829111 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 06:07:32.829271 (+ 160us) server_negotiation.cc:407] Connection header received
0620 06:07:32.829426 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:07:32.829430 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:07:32.829482 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:07:32.829564 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:07:32.829997 (+ 433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:07:32.830549 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:07:32.831450 (+ 901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:07:32.831723 (+ 273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:07:32.831827 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:07:32.834429 (+ 2602us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.102.3:46029: BlockingRecv error: recv got EOF from 127.11.102.3:46029 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
Jun 20 06:07:33 dist-test-slave-9hm5 krb5kdc[19824](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.102.1@KRBTEST.COM for oryx/127.11.102.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 06:07:33 dist-test-slave-9hm5 krb5kdc[19824](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.102.1@KRBTEST.COM for oryx/127.11.102.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 06:07:33.463097 20853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:07:33.457426 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.102.62:43493 (local address 127.11.102.1:44693)
0620 06:07:33.457706 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0620 06:07:33.457721 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 06:07:33.457812 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:07:33.458080 (+ 268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:07:33.458084 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:07:33.458259 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:07:33.458357 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:07:33.458362 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:07:33.459406 (+ 1044us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:07:33.459409 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:07:33.460028 (+ 619us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:07:33.460038 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:07:33.460459 (+ 421us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:07:33.460461 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:07:33.460527 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:07:33.461061 (+ 534us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:07:33.461087 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:07:33.462932 (+ 1845us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.102.62:43493: Server oryx/127.11.102.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":107,"threads_started":1}
W20260620 06:07:33.463172 20854 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 06:07:33.457617 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.102.1:44693 (local address 127.11.102.62:43493)
0620 06:07:33.457820 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0620 06:07:33.457824 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 06:07:33.457837 (+ 13us) server_negotiation.cc:407] Connection header received
0620 06:07:33.457874 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:07:33.457877 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:07:33.457917 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:07:33.457999 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:07:33.458469 (+ 470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:07:33.459259 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:07:33.460154 (+ 895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:07:33.460365 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:07:33.460444 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:07:33.463052 (+ 2608us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.102.1:44693: BlockingRecv error: recv got EOF from 127.11.102.1:44693 (error 108)
Metrics: {"server-negotiator.queue_time_us":147,"thread_start_us":74,"threads_started":1}
Jun 20 06:07:33 dist-test-slave-9hm5 krb5kdc[19824](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.102.2@KRBTEST.COM for oryx/127.11.102.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 06:07:33 dist-test-slave-9hm5 krb5kdc[19824](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.102.2@KRBTEST.COM for oryx/127.11.102.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 06:07:33.636439 20855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:07:33.630811 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.102.62:43493 (local address 127.11.102.2:47145)
0620 06:07:33.631135 (+ 324us) negotiation.cc:107] Waiting for socket to connect
0620 06:07:33.631147 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 06:07:33.631279 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:07:33.631534 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:07:33.631537 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:07:33.631766 (+ 229us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:07:33.631855 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:07:33.631861 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:07:33.632733 (+ 872us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:07:33.632738 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:07:33.633302 (+ 564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:07:33.633309 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:07:33.633777 (+ 468us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:07:33.633780 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:07:33.633867 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:07:33.634375 (+ 508us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:07:33.634386 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:07:33.636272 (+ 1886us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.102.62:43493: Server oryx/127.11.102.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":226,"thread_start_us":84,"threads_started":1}
W20260620 06:07:33.636498 20854 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 06:07:33.630912 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.102.2:47145 (local address 127.11.102.62:43493)
0620 06:07:33.631074 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0620 06:07:33.631082 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 06:07:33.631220 (+ 138us) server_negotiation.cc:407] Connection header received
0620 06:07:33.631374 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:07:33.631379 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:07:33.631428 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:07:33.631511 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:07:33.631967 (+ 456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:07:33.632625 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:07:33.633415 (+ 790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:07:33.633660 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:07:33.633772 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:07:33.636393 (+ 2621us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.102.2:47145: BlockingRecv error: recv got EOF from 127.11.102.2:47145 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
Jun 20 06:07:33 dist-test-slave-9hm5 krb5kdc[19824](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.102.3@KRBTEST.COM for oryx/127.11.102.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 06:07:33 dist-test-slave-9hm5 krb5kdc[19824](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.102.3@KRBTEST.COM for oryx/127.11.102.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 06:07:33.840370 20856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:07:33.835197 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.102.62:43493 (local address 127.11.102.3:34491)
0620 06:07:33.835445 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0620 06:07:33.835460 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 06:07:33.835575 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:07:33.835851 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:07:33.835853 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:07:33.836029 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:07:33.836118 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:07:33.836123 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:07:33.836837 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:07:33.836840 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:07:33.837414 (+ 574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:07:33.837422 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:07:33.837819 (+ 397us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:07:33.837822 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:07:33.837903 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:07:33.838454 (+ 551us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:07:33.838465 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:07:33.840258 (+ 1793us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.102.62:43493: Server oryx/127.11.102.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":175,"thread_start_us":90,"threads_started":1}
W20260620 06:07:33.840493 20854 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 06:07:33.835294 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.102.3:34491 (local address 127.11.102.62:43493)
0620 06:07:33.835449 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0620 06:07:33.835452 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 06:07:33.835557 (+ 105us) server_negotiation.cc:407] Connection header received
0620 06:07:33.835703 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:07:33.835705 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:07:33.835750 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:07:33.835820 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:07:33.836215 (+ 395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:07:33.836721 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:07:33.837519 (+ 798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:07:33.837724 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:07:33.837799 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:07:33.840375 (+ 2576us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.102.3:34491: BlockingRecv error: recv got EOF from 127.11.102.3:34491 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
Jun 20 06:07:34 dist-test-slave-9hm5 krb5kdc[19824](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.102.1@KRBTEST.COM for oryx/127.11.102.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 06:07:34 dist-test-slave-9hm5 krb5kdc[19824](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.102.1@KRBTEST.COM for oryx/127.11.102.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 06:07:34.473605 20857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:07:34.463950 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.102.62:43493 (local address 127.11.102.1:60207)
0620 06:07:34.464255 (+ 305us) negotiation.cc:107] Waiting for socket to connect
0620 06:07:34.464269 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 06:07:34.464355 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:07:34.464639 (+ 284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:07:34.464642 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:07:34.464829 (+ 187us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:07:34.464925 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:07:34.464930 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:07:34.465684 (+ 754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:07:34.465686 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:07:34.466259 (+ 573us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:07:34.466266 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:07:34.466703 (+ 437us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:07:34.466705 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:07:34.466762 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:07:34.471588 (+ 4826us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:07:34.471600 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:07:34.473476 (+ 1876us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.102.62:43493: Server oryx/127.11.102.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":233,"thread_start_us":102,"threads_started":1}
W20260620 06:07:34.473820 20858 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 06:07:34.464059 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.102.1:60207 (local address 127.11.102.62:43493)
0620 06:07:34.464300 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0620 06:07:34.464303 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 06:07:34.464317 (+ 14us) server_negotiation.cc:407] Connection header received
0620 06:07:34.464440 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:07:34.464442 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:07:34.464485 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:07:34.464578 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:07:34.465049 (+ 471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:07:34.465593 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:07:34.466373 (+ 780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:07:34.466605 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:07:34.466687 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:07:34.473639 (+ 6952us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.102.1:60207: BlockingRecv error: recv got EOF from 127.11.102.1:60207 (error 108)
Metrics: {"server-negotiator.queue_time_us":171,"thread_start_us":62,"threads_started":1}
Jun 20 06:07:34 dist-test-slave-9hm5 krb5kdc[19824](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.102.2@KRBTEST.COM for oryx/127.11.102.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 06:07:34 dist-test-slave-9hm5 krb5kdc[19824](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.102.2@KRBTEST.COM for oryx/127.11.102.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 06:07:34.643149 20859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:07:34.637343 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.102.62:43493 (local address 127.11.102.2:58757)
0620 06:07:34.637629 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0620 06:07:34.637644 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 06:07:34.637785 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:07:34.638073 (+ 288us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:07:34.638076 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:07:34.638284 (+ 208us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:07:34.638376 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:07:34.638381 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:07:34.639133 (+ 752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:07:34.639140 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:07:34.639815 (+ 675us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:07:34.639824 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:07:34.640308 (+ 484us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:07:34.640311 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:07:34.640394 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:07:34.640923 (+ 529us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:07:34.640942 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:07:34.642947 (+ 2005us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.102.62:43493: Server oryx/127.11.102.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":206,"thread_start_us":85,"threads_started":1}
W20260620 06:07:34.643329 20858 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 06:07:34.637434 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.102.2:58757 (local address 127.11.102.62:43493)
0620 06:07:34.637605 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0620 06:07:34.637610 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 06:07:34.637729 (+ 119us) server_negotiation.cc:407] Connection header received
0620 06:07:34.637889 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:07:34.637893 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:07:34.637943 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:07:34.638038 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:07:34.638491 (+ 453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:07:34.639024 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:07:34.639926 (+ 902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:07:34.640149 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:07:34.640301 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:07:34.643189 (+ 2888us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.102.2:58757: BlockingRecv error: recv got EOF from 127.11.102.2:58757 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
Jun 20 06:07:34 dist-test-slave-9hm5 krb5kdc[19824](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.102.3@KRBTEST.COM for oryx/127.11.102.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 06:07:34 dist-test-slave-9hm5 krb5kdc[19824](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.102.3@KRBTEST.COM for oryx/127.11.102.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 06:07:34.846478 20860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:07:34.841093 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.102.62:43493 (local address 127.11.102.3:46725)
0620 06:07:34.841358 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0620 06:07:34.841373 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 06:07:34.841474 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:07:34.841681 (+ 207us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:07:34.841683 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:07:34.841873 (+ 190us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:07:34.841965 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:07:34.841970 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:07:34.842783 (+ 813us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:07:34.842786 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:07:34.843521 (+ 735us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:07:34.843531 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:07:34.843997 (+ 466us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:07:34.844000 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:07:34.844064 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:07:34.844546 (+ 482us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:07:34.844557 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:07:34.846363 (+ 1806us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.102.62:43493: Server oryx/127.11.102.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":82,"threads_started":1}
W20260620 06:07:34.846657 20858 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 06:07:34.841225 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.102.3:46725 (local address 127.11.102.62:43493)
0620 06:07:34.841376 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0620 06:07:34.841379 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 06:07:34.841425 (+ 46us) server_negotiation.cc:407] Connection header received
0620 06:07:34.841540 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:07:34.841542 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:07:34.841585 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:07:34.841650 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:07:34.842055 (+ 405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:07:34.842683 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:07:34.843661 (+ 978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:07:34.843893 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:07:34.843987 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:07:34.846484 (+ 2497us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.102.3:46725: BlockingRecv error: recv got EOF from 127.11.102.3:46725 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
Jun 20 06:07:35 dist-test-slave-9hm5 krb5kdc[19824](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.102.1@KRBTEST.COM for oryx/127.11.102.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 06:07:35 dist-test-slave-9hm5 krb5kdc[19824](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.102.1@KRBTEST.COM for oryx/127.11.102.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 06:07:35.480196 20861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:07:35.474418 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.102.62:43493 (local address 127.11.102.1:49315)
0620 06:07:35.474706 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0620 06:07:35.474720 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 06:07:35.474852 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:07:35.475205 (+ 353us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:07:35.475208 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:07:35.475451 (+ 243us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:07:35.475590 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:07:35.475599 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:07:35.476610 (+ 1011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:07:35.476612 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:07:35.477168 (+ 556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:07:35.477175 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:07:35.477614 (+ 439us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:07:35.477617 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:07:35.477679 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:07:35.478229 (+ 550us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:07:35.478243 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:07:35.480071 (+ 1828us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.102.62:43493: Server oryx/127.11.102.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":197,"thread_start_us":111,"threads_started":1}
W20260620 06:07:35.480381 20862 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 06:07:35.474524 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.102.1:49315 (local address 127.11.102.62:43493)
0620 06:07:35.474757 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0620 06:07:35.474760 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 06:07:35.474773 (+ 13us) server_negotiation.cc:407] Connection header received
0620 06:07:35.474954 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:07:35.474957 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:07:35.475021 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:07:35.475139 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:07:35.475753 (+ 614us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:07:35.476515 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:07:35.477288 (+ 773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:07:35.477509 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:07:35.477639 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:07:35.480206 (+ 2567us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.102.1:49315: BlockingRecv error: recv got EOF from 127.11.102.1:49315 (error 108)
Metrics: {"server-negotiator.queue_time_us":161,"thread_start_us":85,"threads_started":1}
Jun 20 06:07:35 dist-test-slave-9hm5 krb5kdc[19824](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.102.2@KRBTEST.COM for oryx/127.11.102.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 06:07:35 dist-test-slave-9hm5 krb5kdc[19824](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.102.2@KRBTEST.COM for oryx/127.11.102.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 06:07:35.653198 20863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:07:35.643923 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.102.62:43493 (local address 127.11.102.2:53979)
0620 06:07:35.646815 (+ 2892us) negotiation.cc:107] Waiting for socket to connect
0620 06:07:35.646832 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 06:07:35.646960 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:07:35.647308 (+ 348us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:07:35.647315 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:07:35.647549 (+ 234us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:07:35.647693 (+ 144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:07:35.647704 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:07:35.648858 (+ 1154us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:07:35.648860 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:07:35.649637 (+ 777us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:07:35.649649 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:07:35.650275 (+ 626us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:07:35.650281 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:07:35.650374 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:07:35.650986 (+ 612us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:07:35.651002 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:07:35.653089 (+ 2087us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.102.62:43493: Server oryx/127.11.102.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":2810,"thread_start_us":83,"threads_started":1}
W20260620 06:07:35.653487 20862 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 06:07:35.643982 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.102.2:53979 (local address 127.11.102.62:43493)
0620 06:07:35.644123 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0620 06:07:35.644126 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 06:07:35.646901 (+ 2775us) server_negotiation.cc:407] Connection header received
0620 06:07:35.647109 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:07:35.647113 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:07:35.647167 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:07:35.647250 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:07:35.647900 (+ 650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:07:35.648750 (+ 850us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:07:35.649774 (+ 1024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:07:35.650133 (+ 359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:07:35.650265 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:07:35.653212 (+ 2947us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.102.2:53979: BlockingRecv error: recv got EOF from 127.11.102.2:53979 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Jun 20 06:07:35 dist-test-slave-9hm5 krb5kdc[19824](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.102.3@KRBTEST.COM for oryx/127.11.102.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 06:07:35 dist-test-slave-9hm5 krb5kdc[19824](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.102.3@KRBTEST.COM for oryx/127.11.102.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 06:07:35.852442 20864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:07:35.847217 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.102.62:43493 (local address 127.11.102.3:43607)
0620 06:07:35.847539 (+ 322us) negotiation.cc:107] Waiting for socket to connect
0620 06:07:35.847552 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 06:07:35.847692 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:07:35.847943 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:07:35.847946 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:07:35.848115 (+ 169us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:07:35.848208 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:07:35.848213 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:07:35.848916 (+ 703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:07:35.848918 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:07:35.849474 (+ 556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:07:35.849483 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:07:35.849870 (+ 387us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:07:35.849873 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:07:35.849939 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:07:35.850437 (+ 498us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:07:35.850448 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:07:35.852272 (+ 1824us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.102.62:43493: Server oryx/127.11.102.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":250,"thread_start_us":148,"threads_started":1}
W20260620 06:07:35.852478 20862 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 06:07:35.847379 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.102.3:43607 (local address 127.11.102.62:43493)
0620 06:07:35.847513 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0620 06:07:35.847517 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 06:07:35.847643 (+ 126us) server_negotiation.cc:407] Connection header received
0620 06:07:35.847776 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:07:35.847778 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:07:35.847820 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:07:35.847909 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:07:35.848301 (+ 392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:07:35.848811 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:07:35.849572 (+ 761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:07:35.849776 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:07:35.849868 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:07:35.852382 (+ 2514us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.102.3:43607: BlockingRecv error: recv got EOF from 127.11.102.3:43607 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
Jun 20 06:07:36 dist-test-slave-9hm5 krb5kdc[19824](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.102.1@KRBTEST.COM for oryx/127.11.102.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 06:07:36 dist-test-slave-9hm5 krb5kdc[19824](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.102.1@KRBTEST.COM for oryx/127.11.102.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 06:07:36.485994 20865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:07:36.480910 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.102.62:43493 (local address 127.11.102.1:50295)
0620 06:07:36.481204 (+ 294us) negotiation.cc:107] Waiting for socket to connect
0620 06:07:36.481218 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 06:07:36.481306 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:07:36.481550 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:07:36.481552 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:07:36.481730 (+ 178us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:07:36.481824 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:07:36.481829 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:07:36.482555 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:07:36.482558 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:07:36.483121 (+ 563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:07:36.483129 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:07:36.483525 (+ 396us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:07:36.483527 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:07:36.483588 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:07:36.484078 (+ 490us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:07:36.484103 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:07:36.485879 (+ 1776us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.102.62:43493: Server oryx/127.11.102.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":199,"thread_start_us":85,"threads_started":1}
W20260620 06:07:36.486209 20866 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 06:07:36.481016 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.102.1:50295 (local address 127.11.102.62:43493)
0620 06:07:36.481257 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0620 06:07:36.481260 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 06:07:36.481273 (+ 13us) server_negotiation.cc:407] Connection header received
0620 06:07:36.481391 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:07:36.481393 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:07:36.481437 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:07:36.481513 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:07:36.481932 (+ 419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:07:36.482458 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:07:36.483222 (+ 764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:07:36.483435 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:07:36.483510 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:07:36.486024 (+ 2514us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.102.1:50295: BlockingRecv error: recv got EOF from 127.11.102.1:50295 (error 108)
Metrics: {"server-negotiator.queue_time_us":177,"thread_start_us":81,"threads_started":1}
Jun 20 06:07:36 dist-test-slave-9hm5 krb5kdc[19824](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.102.2@KRBTEST.COM for oryx/127.11.102.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 06:07:36 dist-test-slave-9hm5 krb5kdc[19824](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.102.2@KRBTEST.COM for oryx/127.11.102.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 06:07:36.659518 20867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:07:36.653912 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.102.62:43493 (local address 127.11.102.2:50465)
0620 06:07:36.654206 (+ 294us) negotiation.cc:107] Waiting for socket to connect
0620 06:07:36.654226 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 06:07:36.654357 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:07:36.654607 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:07:36.654610 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:07:36.654817 (+ 207us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:07:36.654909 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:07:36.654915 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:07:36.655830 (+ 915us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:07:36.655834 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:07:36.656407 (+ 573us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:07:36.656415 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:07:36.656858 (+ 443us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:07:36.656862 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:07:36.656952 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:07:36.657475 (+ 523us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:07:36.657490 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:07:36.659327 (+ 1837us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.102.62:43493: Server oryx/127.11.102.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":216,"thread_start_us":84,"threads_started":1}
W20260620 06:07:36.659560 20866 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 06:07:36.654014 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.102.2:50465 (local address 127.11.102.62:43493)
0620 06:07:36.654167 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0620 06:07:36.654171 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 06:07:36.654298 (+ 127us) server_negotiation.cc:407] Connection header received
0620 06:07:36.654453 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:07:36.654456 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:07:36.654504 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:07:36.654588 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:07:36.655023 (+ 435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:07:36.655534 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:07:36.656529 (+ 995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:07:36.656750 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:07:36.656855 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:07:36.659446 (+ 2591us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.102.2:50465: BlockingRecv error: recv got EOF from 127.11.102.2:50465 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
Jun 20 06:07:36 dist-test-slave-9hm5 krb5kdc[19824](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.102.3@KRBTEST.COM for oryx/127.11.102.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 06:07:36 dist-test-slave-9hm5 krb5kdc[19824](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.102.3@KRBTEST.COM for oryx/127.11.102.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 06:07:36.858994 20868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:07:36.853159 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.102.62:43493 (local address 127.11.102.3:38779)
0620 06:07:36.853480 (+ 321us) negotiation.cc:107] Waiting for socket to connect
0620 06:07:36.853494 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 06:07:36.853598 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:07:36.853825 (+ 227us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:07:36.853827 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:07:36.854018 (+ 191us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:07:36.854118 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:07:36.854123 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:07:36.854846 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:07:36.854849 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:07:36.855461 (+ 612us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:07:36.855470 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:07:36.856033 (+ 563us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:07:36.856053 (+ 20us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:07:36.856153 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:07:36.856797 (+ 644us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:07:36.856812 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:07:36.858800 (+ 1988us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.102.62:43493: Server oryx/127.11.102.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":222,"thread_start_us":127,"threads_started":1}
W20260620 06:07:36.859018 20866 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 06:07:36.853236 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.102.3:38779 (local address 127.11.102.62:43493)
0620 06:07:36.853359 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0620 06:07:36.853362 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 06:07:36.853590 (+ 228us) server_negotiation.cc:407] Connection header received
0620 06:07:36.853669 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:07:36.853671 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:07:36.853720 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:07:36.853794 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:07:36.854221 (+ 427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:07:36.854753 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:07:36.855597 (+ 844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:07:36.855921 (+ 324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:07:36.856024 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:07:36.858916 (+ 2892us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.102.3:38779: BlockingRecv error: recv got EOF from 127.11.102.3:38779 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
Jun 20 06:07:37 dist-test-slave-9hm5 krb5kdc[19824](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.102.1@KRBTEST.COM for oryx/127.11.102.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 06:07:37 dist-test-slave-9hm5 krb5kdc[19824](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.102.1@KRBTEST.COM for oryx/127.11.102.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 06:07:37.492277 20869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:07:37.486747 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.102.62:43493 (local address 127.11.102.1:57421)
0620 06:07:37.487030 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0620 06:07:37.487044 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 06:07:37.487139 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:07:37.487418 (+ 279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:07:37.487421 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:07:37.487637 (+ 216us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:07:37.487740 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:07:37.487746 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:07:37.488570 (+ 824us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:07:37.488573 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:07:37.489136 (+ 563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:07:37.489144 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:07:37.489650 (+ 506us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:07:37.489653 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:07:37.489729 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:07:37.490300 (+ 571us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:07:37.490314 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:07:37.492166 (+ 1852us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.102.62:43493: Server oryx/127.11.102.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":210,"thread_start_us":103,"threads_started":1}
W20260620 06:07:37.492452 20870 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 06:07:37.486819 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.102.1:57421 (local address 127.11.102.62:43493)
0620 06:07:37.487045 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0620 06:07:37.487050 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 06:07:37.487110 (+ 60us) server_negotiation.cc:407] Connection header received
0620 06:07:37.487230 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:07:37.487232 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:07:37.487277 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:07:37.487352 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:07:37.487891 (+ 539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:07:37.488421 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:07:37.489254 (+ 833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:07:37.489463 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:07:37.489536 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:07:37.492290 (+ 2754us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.102.1:57421: BlockingRecv error: recv got EOF from 127.11.102.1:57421 (error 108)
Metrics: {"server-negotiator.queue_time_us":156,"thread_start_us":64,"threads_started":1}
Jun 20 06:07:37 dist-test-slave-9hm5 krb5kdc[19824](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.102.2@KRBTEST.COM for oryx/127.11.102.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 06:07:37 dist-test-slave-9hm5 krb5kdc[19824](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.102.2@KRBTEST.COM for oryx/127.11.102.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 06:07:37.665694 20871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:07:37.660260 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.102.62:43493 (local address 127.11.102.2:55879)
0620 06:07:37.660582 (+ 322us) negotiation.cc:107] Waiting for socket to connect
0620 06:07:37.660599 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 06:07:37.660738 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:07:37.661019 (+ 281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:07:37.661022 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:07:37.661225 (+ 203us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:07:37.661316 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:07:37.661322 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:07:37.662045 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:07:37.662049 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:07:37.662604 (+ 555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:07:37.662611 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:07:37.663044 (+ 433us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:07:37.663048 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:07:37.663129 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:07:37.663711 (+ 582us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:07:37.663724 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:07:37.665584 (+ 1860us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.102.62:43493: Server oryx/127.11.102.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":236,"thread_start_us":115,"threads_started":1}
W20260620 06:07:37.665860 20870 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 06:07:37.660426 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.102.2:55879 (local address 127.11.102.62:43493)
0620 06:07:37.660579 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0620 06:07:37.660584 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 06:07:37.660676 (+ 92us) server_negotiation.cc:407] Connection header received
0620 06:07:37.660869 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:07:37.660872 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:07:37.660922 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:07:37.661001 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:07:37.661428 (+ 427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:07:37.661939 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:07:37.662717 (+ 778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:07:37.662937 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:07:37.663041 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:07:37.665711 (+ 2670us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.102.2:55879: BlockingRecv error: recv got EOF from 127.11.102.2:55879 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
Jun 20 06:07:37 dist-test-slave-9hm5 krb5kdc[19824](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.102.3@KRBTEST.COM for oryx/127.11.102.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 06:07:37 dist-test-slave-9hm5 krb5kdc[19824](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.102.3@KRBTEST.COM for oryx/127.11.102.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 06:07:37.865193 20872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:07:37.859792 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.102.62:43493 (local address 127.11.102.3:48131)
0620 06:07:37.860050 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0620 06:07:37.860063 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 06:07:37.860160 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:07:37.860431 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:07:37.860434 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:07:37.860625 (+ 191us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:07:37.860724 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:07:37.860729 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:07:37.861507 (+ 778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:07:37.861510 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:07:37.862055 (+ 545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:07:37.862062 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:07:37.862489 (+ 427us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:07:37.862492 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:07:37.862550 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:07:37.863121 (+ 571us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:07:37.863132 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:07:37.865071 (+ 1939us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.102.62:43493: Server oryx/127.11.102.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":92,"threads_started":1}
W20260620 06:07:37.865312 20870 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 06:07:37.859854 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.102.3:48131 (local address 127.11.102.62:43493)
0620 06:07:37.860008 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0620 06:07:37.860012 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 06:07:37.860130 (+ 118us) server_negotiation.cc:407] Connection header received
0620 06:07:37.860238 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:07:37.860240 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:07:37.860287 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:07:37.860353 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:07:37.860842 (+ 489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:07:37.861412 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:07:37.862155 (+ 743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:07:37.862381 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:07:37.862472 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:07:37.865182 (+ 2710us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.102.3:48131: BlockingRecv error: recv got EOF from 127.11.102.3:48131 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
Jun 20 06:07:38 dist-test-slave-9hm5 krb5kdc[19824](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.102.1@KRBTEST.COM for oryx/127.11.102.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 06:07:38 dist-test-slave-9hm5 krb5kdc[19824](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.102.1@KRBTEST.COM for oryx/127.11.102.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 06:07:38.498164 20873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:07:38.493004 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.102.62:43493 (local address 127.11.102.1:45405)
0620 06:07:38.493259 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0620 06:07:38.493272 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 06:07:38.493366 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:07:38.493565 (+ 199us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:07:38.493567 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:07:38.493740 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:07:38.493823 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:07:38.493827 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:07:38.494459 (+ 632us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:07:38.494461 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:07:38.494982 (+ 521us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:07:38.494988 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:07:38.495410 (+ 422us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:07:38.495414 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:07:38.495551 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:07:38.496152 (+ 601us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:07:38.496166 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:07:38.498032 (+ 1866us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.102.62:43493: Server oryx/127.11.102.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":90,"threads_started":1}
W20260620 06:07:38.498325 20874 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 06:07:38.493061 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.102.1:45405 (local address 127.11.102.62:43493)
0620 06:07:38.493283 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0620 06:07:38.493286 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 06:07:38.493332 (+ 46us) server_negotiation.cc:407] Connection header received
0620 06:07:38.493436 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:07:38.493438 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:07:38.493478 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:07:38.493556 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:07:38.493901 (+ 345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:07:38.494380 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:07:38.495067 (+ 687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:07:38.495295 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:07:38.495398 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:07:38.498162 (+ 2764us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.102.1:45405: BlockingRecv error: recv got EOF from 127.11.102.1:45405 (error 108)
Metrics: {"server-negotiator.queue_time_us":161,"thread_start_us":88,"threads_started":1}
Jun 20 06:07:38 dist-test-slave-9hm5 krb5kdc[19824](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.102.2@KRBTEST.COM for oryx/127.11.102.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 06:07:38 dist-test-slave-9hm5 krb5kdc[19824](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.102.2@KRBTEST.COM for oryx/127.11.102.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 06:07:38.671743 20875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:07:38.666523 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.102.62:43493 (local address 127.11.102.2:53001)
0620 06:07:38.666800 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0620 06:07:38.666818 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 06:07:38.666966 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:07:38.667264 (+ 298us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:07:38.667267 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:07:38.667511 (+ 244us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:07:38.667653 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:07:38.667661 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:07:38.668418 (+ 757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:07:38.668421 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:07:38.668970 (+ 549us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:07:38.668978 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:07:38.669381 (+ 403us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:07:38.669383 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:07:38.669440 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:07:38.669909 (+ 469us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:07:38.669919 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:07:38.671575 (+ 1656us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.102.62:43493: Server oryx/127.11.102.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":100,"threads_started":1}
W20260620 06:07:38.671856 20874 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 06:07:38.666694 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.102.2:53001 (local address 127.11.102.62:43493)
0620 06:07:38.666806 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0620 06:07:38.666813 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 06:07:38.666907 (+ 94us) server_negotiation.cc:407] Connection header received
0620 06:07:38.667075 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:07:38.667078 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:07:38.667127 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:07:38.667212 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:07:38.667758 (+ 546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:07:38.668326 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:07:38.669087 (+ 761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:07:38.669292 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:07:38.669366 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:07:38.671749 (+ 2383us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.102.2:53001: BlockingRecv error: recv got EOF from 127.11.102.2:53001 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
Jun 20 06:07:38 dist-test-slave-9hm5 krb5kdc[19824](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.102.3@KRBTEST.COM for oryx/127.11.102.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 06:07:38 dist-test-slave-9hm5 krb5kdc[19824](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.102.3@KRBTEST.COM for oryx/127.11.102.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 06:07:38.870924 20876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:07:38.865866 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.102.62:43493 (local address 127.11.102.3:36365)
0620 06:07:38.866142 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0620 06:07:38.866155 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 06:07:38.866247 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:07:38.866457 (+ 210us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:07:38.866459 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:07:38.866636 (+ 177us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:07:38.866727 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:07:38.866732 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:07:38.867465 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:07:38.867468 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:07:38.868039 (+ 571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:07:38.868047 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:07:38.868484 (+ 437us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:07:38.868487 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:07:38.868551 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:07:38.869005 (+ 454us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:07:38.869016 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:07:38.870814 (+ 1798us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.102.62:43493: Server oryx/127.11.102.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":115,"threads_started":1}
W20260620 06:07:38.871189 20874 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 06:07:38.865931 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.102.3:36365 (local address 127.11.102.62:43493)
0620 06:07:38.866056 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0620 06:07:38.866059 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 06:07:38.866221 (+ 162us) server_negotiation.cc:407] Connection header received
0620 06:07:38.866310 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:07:38.866312 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:07:38.866353 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:07:38.866421 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:07:38.866824 (+ 403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:07:38.867355 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:07:38.868142 (+ 787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:07:38.868351 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:07:38.868427 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:07:38.870962 (+ 2535us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.102.3:36365: BlockingRecv error: recv got EOF from 127.11.102.3:36365 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
Jun 20 06:07:39 dist-test-slave-9hm5 krb5kdc[19824](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.102.1@KRBTEST.COM for oryx/127.11.102.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 06:07:39 dist-test-slave-9hm5 krb5kdc[19824](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.102.1@KRBTEST.COM for oryx/127.11.102.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 06:07:39.508657 20877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:07:39.498918 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.102.62:43493 (local address 127.11.102.1:45179)
0620 06:07:39.499196 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0620 06:07:39.499210 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 06:07:39.499297 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:07:39.499644 (+ 347us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:07:39.499646 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:07:39.499828 (+ 182us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:07:39.499921 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:07:39.499926 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:07:39.500670 (+ 744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:07:39.500673 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:07:39.501225 (+ 552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:07:39.501232 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:07:39.501620 (+ 388us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:07:39.501623 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:07:39.501694 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:07:39.506551 (+ 4857us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:07:39.506563 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:07:39.508472 (+ 1909us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.102.62:43493: Server oryx/127.11.102.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":199,"thread_start_us":84,"threads_started":1}
W20260620 06:07:39.508708 20878 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 06:07:39.498976 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.102.1:45179 (local address 127.11.102.62:43493)
0620 06:07:39.499255 (+ 279us) server_negotiation.cc:206] Beginning negotiation
0620 06:07:39.499258 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 06:07:39.499271 (+ 13us) server_negotiation.cc:407] Connection header received
0620 06:07:39.499463 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:07:39.499466 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:07:39.499513 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:07:39.499600 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:07:39.500031 (+ 431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:07:39.500568 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:07:39.501320 (+ 752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:07:39.501528 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:07:39.501606 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:07:39.508585 (+ 6979us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.102.1:45179: BlockingRecv error: recv got EOF from 127.11.102.1:45179 (error 108)
Metrics: {"server-negotiator.queue_time_us":216,"thread_start_us":54,"threads_started":1}
Jun 20 06:07:39 dist-test-slave-9hm5 krb5kdc[19824](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.102.2@KRBTEST.COM for oryx/127.11.102.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 06:07:39 dist-test-slave-9hm5 krb5kdc[19824](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.102.2@KRBTEST.COM for oryx/127.11.102.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 06:07:39.681910 20879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:07:39.672530 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.102.62:43493 (local address 127.11.102.2:48819)
0620 06:07:39.672855 (+ 325us) negotiation.cc:107] Waiting for socket to connect
0620 06:07:39.672878 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0620 06:07:39.673013 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:07:39.673305 (+ 292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:07:39.673307 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:07:39.673514 (+ 207us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:07:39.673608 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:07:39.673614 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:07:39.674350 (+ 736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:07:39.674356 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:07:39.674940 (+ 584us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:07:39.674949 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:07:39.675387 (+ 438us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:07:39.675390 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:07:39.675480 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:07:39.679831 (+ 4351us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:07:39.679844 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:07:39.681782 (+ 1938us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.102.62:43493: Server oryx/127.11.102.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":231,"thread_start_us":90,"threads_started":1}
W20260620 06:07:39.682049 20878 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 06:07:39.672649 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.102.2:48819 (local address 127.11.102.62:43493)
0620 06:07:39.672830 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0620 06:07:39.672834 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 06:07:39.672954 (+ 120us) server_negotiation.cc:407] Connection header received
0620 06:07:39.673147 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:07:39.673150 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:07:39.673200 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:07:39.673289 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:07:39.673724 (+ 435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:07:39.674243 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:07:39.675059 (+ 816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:07:39.675277 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:07:39.675384 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:07:39.681925 (+ 6541us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.102.2:48819: BlockingRecv error: recv got EOF from 127.11.102.2:48819 (error 108)
Metrics: {"server-negotiator.queue_time_us":75}
Jun 20 06:07:39 dist-test-slave-9hm5 krb5kdc[19824](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.102.3@KRBTEST.COM for oryx/127.11.102.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 06:07:39 dist-test-slave-9hm5 krb5kdc[19824](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.102.3@KRBTEST.COM for oryx/127.11.102.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 06:07:39.876955 20880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:07:39.871653 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.102.62:43493 (local address 127.11.102.3:40153)
0620 06:07:39.871945 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0620 06:07:39.871957 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 06:07:39.872086 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:07:39.872342 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:07:39.872345 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:07:39.872564 (+ 219us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:07:39.872652 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:07:39.872657 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:07:39.873385 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:07:39.873390 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:07:39.873945 (+ 555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:07:39.873953 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:07:39.874388 (+ 435us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:07:39.874392 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:07:39.874474 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:07:39.874935 (+ 461us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:07:39.874945 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:07:39.876791 (+ 1846us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.102.62:43493: Server oryx/127.11.102.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":223,"thread_start_us":72,"threads_started":1}
W20260620 06:07:39.876996 20878 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 06:07:39.871730 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.102.3:40153 (local address 127.11.102.62:43493)
0620 06:07:39.871863 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0620 06:07:39.871867 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 06:07:39.872033 (+ 166us) server_negotiation.cc:407] Connection header received
0620 06:07:39.872184 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:07:39.872189 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:07:39.872239 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:07:39.872320 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:07:39.872761 (+ 441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:07:39.873279 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:07:39.874064 (+ 785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:07:39.874280 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:07:39.874386 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:07:39.876899 (+ 2513us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.102.3:40153: BlockingRecv error: recv got EOF from 127.11.102.3:40153 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
Jun 20 06:07:40 dist-test-slave-9hm5 krb5kdc[19824](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.102.1@KRBTEST.COM for oryx/127.11.102.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 06:07:40 dist-test-slave-9hm5 krb5kdc[19824](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.102.1@KRBTEST.COM for oryx/127.11.102.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 06:07:40.515381 20881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:07:40.509596 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.102.62:43493 (local address 127.11.102.1:44691)
0620 06:07:40.509876 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0620 06:07:40.509890 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 06:07:40.509973 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:07:40.510213 (+ 240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:07:40.510215 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:07:40.510386 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:07:40.510477 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:07:40.510482 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:07:40.511240 (+ 758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:07:40.511243 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:07:40.511913 (+ 670us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:07:40.511922 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:07:40.512381 (+ 459us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:07:40.512384 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:07:40.512442 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:07:40.512924 (+ 482us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:07:40.512936 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:07:40.515208 (+ 2272us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.102.62:43493: Server oryx/127.11.102.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":78,"threads_started":1}
W20260620 06:07:40.515512 20882 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 06:07:40.509665 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.102.1:44691 (local address 127.11.102.62:43493)
0620 06:07:40.509939 (+ 274us) server_negotiation.cc:206] Beginning negotiation
0620 06:07:40.509943 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 06:07:40.509958 (+ 15us) server_negotiation.cc:407] Connection header received
0620 06:07:40.510055 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:07:40.510059 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:07:40.510105 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:07:40.510189 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:07:40.510591 (+ 402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:07:40.511121 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:07:40.512040 (+ 919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:07:40.512280 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:07:40.512368 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:07:40.515370 (+ 3002us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.102.1:44691: BlockingRecv error: recv got EOF from 127.11.102.1:44691 (error 108)
Metrics: {"server-negotiator.queue_time_us":204,"thread_start_us":102,"threads_started":1}
Jun 20 06:07:40 dist-test-slave-9hm5 krb5kdc[19824](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.102.2@KRBTEST.COM for oryx/127.11.102.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 06:07:40 dist-test-slave-9hm5 krb5kdc[19824](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.11.102.2@KRBTEST.COM for oryx/127.11.102.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 06:07:40.688243 20883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:07:40.682821 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.102.62:43493 (local address 127.11.102.2:51863)
0620 06:07:40.683112 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0620 06:07:40.683129 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 06:07:40.683262 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:07:40.683544 (+ 282us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:07:40.683547 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:07:40.683766 (+ 219us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:07:40.683862 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:07:40.683867 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:07:40.684601 (+ 734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:07:40.684607 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:07:40.685145 (+ 538us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:07:40.685152 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:07:40.685586 (+ 434us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:07:40.685590 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:07:40.685673 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:07:40.686237 (+ 564us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:07:40.686248 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:07:40.688124 (+ 1876us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.11.102.62:43493: Server oryx/127.11.102.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":219,"thread_start_us":77,"threads_started":1}
W20260620 06:07:40.688398 20882 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 06:07:40.682948 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.102.2:51863 (local address 127.11.102.62:43493)
0620 06:07:40.683114 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0620 06:07:40.683120 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 06:07:40.683209 (+ 89us) server_negotiation.cc:407] Connection header received
0620 06:07:40.683356 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:07:40.683361 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:07:40.683408 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:07:40.683497 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:07:40.683977 (+ 480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:07:40.684499 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:07:40.685255 (+ 756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:07:40.685482 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:07:40.685583 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:07:40.688251 (+ 2668us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.11.102.2:51863: BlockingRecv error: recv got EOF from 127.11.102.2:51863 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
I20260620 06:07:40.769910 11672 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9HMp43/build/debug/bin/kudu with pid 20407
I20260620 06:07:40.775321 11672 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9HMp43/build/debug/bin/kudu with pid 20559
I20260620 06:07:40.780790 11672 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9HMp43/build/debug/bin/kudu with pid 20691
I20260620 06:07:40.785868 11672 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9HMp43/build/debug/bin/kudu with pid 20337
2026-06-20T06:07:40Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (17928 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-06-20T06:07:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T06:07:40Z Disabled control of system clock
I20260620 06:07:40.815591 11672 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9HMp43/build/debug/bin/kudu
/tmp/dist-test-task9HMp43/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781935603120653-11672-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781935603120653-11672-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781935603120653-11672-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781935603120653-11672-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.102.62:45915
--webserver_interface=127.11.102.62
--webserver_port=0
--builtin_ntp_servers=127.11.102.20:33047
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.102.62:45915
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260620 06:07:40.923182 20887 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:07:40.923461 20887 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:07:40.923501 20887 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:07:40.927464 20887 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 06:07:40.927524 20887 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:07:40.927541 20887 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:07:40.927557 20887 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 06:07:40.927573 20887 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 06:07:40.932299 20887 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.102.20:33047
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781935603120653-11672-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781935603120653-11672-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.102.62:45915
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.102.62:45915
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781935603120653-11672-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.102.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20887
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781935603120653-11672-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 05:28:48 UTC on ccycloud.quasar-fixhms.root.comops.site
build id 4457020
I20260620 06:07:40.933389 20887 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:07:40.934698 20887 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 06:07:40.943416 20893 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:07:40.943440 20892 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:07:40.943574 20895 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:07:40.943512 20887 server_base.cc:1034] running on GCE node
I20260620 06:07:40.944175 20887 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:07:40.945231 20887 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:07:40.946441 20887 hybrid_clock.cc:648] HybridClock initialized: now 1781935660946427 us; error 27 us; skew 500 ppm
I20260620 06:07:40.948329 20887 webserver.cc:460] Webserver started at http://127.11.102.62:39613/ using document root <none> and password file <none>
I20260620 06:07:40.948904 20887 fs_manager.cc:359] Metadata directory not provided
I20260620 06:07:40.948992 20887 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:07:40.949210 20887 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 06:07:40.951169 20887 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781935603120653-11672-0/minicluster-data/master-0/data/instance:
uuid: "e5c3f1dad6bb496bac7393d86f82278f"
format_stamp: "Formatted at 2026-06-20 06:07:40 on dist-test-slave-9hm5"
server_key: "112298159f2f1b6788ad6ce1db5de425"
server_key_iv: "28eb582d56b61ca5653fc9de6052836a"
server_key_version: "encryptionkey@0"
I20260620 06:07:40.951761 20887 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781935603120653-11672-0/minicluster-data/master-0/wal/instance:
uuid: "e5c3f1dad6bb496bac7393d86f82278f"
format_stamp: "Formatted at 2026-06-20 06:07:40 on dist-test-slave-9hm5"
server_key: "112298159f2f1b6788ad6ce1db5de425"
server_key_iv: "28eb582d56b61ca5653fc9de6052836a"
server_key_version: "encryptionkey@0"
I20260620 06:07:40.955184 20887 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260620 06:07:40.957484 20901 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:07:40.958564 20887 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 06:07:40.958691 20887 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781935603120653-11672-0/minicluster-data/master-0/data,/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781935603120653-11672-0/minicluster-data/master-0/wal
uuid: "e5c3f1dad6bb496bac7393d86f82278f"
format_stamp: "Formatted at 2026-06-20 06:07:40 on dist-test-slave-9hm5"
server_key: "112298159f2f1b6788ad6ce1db5de425"
server_key_iv: "28eb582d56b61ca5653fc9de6052836a"
server_key_version: "encryptionkey@0"
I20260620 06:07:40.958792 20887 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781935603120653-11672-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781935603120653-11672-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781935603120653-11672-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:07:40.970510 20887 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:07:40.971081 20887 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:07:40.971302 20887 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:07:40.977710 20887 rpc_server.cc:307] RPC server started. Bound to: 127.11.102.62:45915
I20260620 06:07:40.977733 20953 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.102.62:45915 every 8 connection(s)
I20260620 06:07:40.978578 20887 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781935603120653-11672-0/minicluster-data/master-0/data/info.pb
I20260620 06:07:40.980733 11672 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9HMp43/build/debug/bin/kudu as pid 20887
I20260620 06:07:40.980841 11672 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781935603120653-11672-0/minicluster-data/master-0/wal/instance
I20260620 06:07:40.981184 11672 external_mini_cluster.cc:1428] Setting key 3b08b23fb505314da28746cbf177ce0f
I20260620 06:07:40.982106 20954 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 06:07:40.988826 20954 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e5c3f1dad6bb496bac7393d86f82278f: Bootstrap starting.
I20260620 06:07:40.991401 20954 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e5c3f1dad6bb496bac7393d86f82278f: Neither blocks nor log segments found. Creating new log.
I20260620 06:07:40.991443 20957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:07:40.982541 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40182 (local address 127.11.102.62:45915)
0620 06:07:40.983189 (+ 648us) server_negotiation.cc:206] Beginning negotiation
0620 06:07:40.983208 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0620 06:07:40.983250 (+ 42us) server_negotiation.cc:407] Connection header received
0620 06:07:40.984168 (+ 918us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:07:40.984190 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:07:40.984689 (+ 499us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:07:40.985052 (+ 363us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:07:40.985489 (+ 437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:07:40.986944 (+ 1455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:07:40.987849 (+ 905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:07:40.988515 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:07:40.988668 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:07:40.989493 (+ 825us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:07:40.989500 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:07:40.989514 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 06:07:40.989998 (+ 484us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:07:40.990067 (+ 69us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 06:07:40.990096 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:07:40.990196 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0620 06:07:40.990286 (+ 90us) server_negotiation.cc:299] Negotiation successful
0620 06:07:40.990545 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":257,"thread_start_us":111,"threads_started":1}
I20260620 06:07:40.992269 20954 log.cc:824] T 00000000000000000000000000000000 P e5c3f1dad6bb496bac7393d86f82278f: Log is configured to *not* fsync() on all Append() calls
I20260620 06:07:40.994132 20954 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e5c3f1dad6bb496bac7393d86f82278f: No bootstrap required, opened a new log
I20260620 06:07:40.997498 20954 raft_consensus.cc:348] T 00000000000000000000000000000000 P e5c3f1dad6bb496bac7393d86f82278f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e5c3f1dad6bb496bac7393d86f82278f" member_type: VOTER last_known_addr { host: "127.11.102.62" port: 45915 } }
I20260620 06:07:40.997695 20954 raft_consensus.cc:374] T 00000000000000000000000000000000 P e5c3f1dad6bb496bac7393d86f82278f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 06:07:40.997728 20954 raft_consensus.cc:729] T 00000000000000000000000000000000 P e5c3f1dad6bb496bac7393d86f82278f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e5c3f1dad6bb496bac7393d86f82278f, State: Initialized, Role: FOLLOWER
I20260620 06:07:40.998071 20954 consensus_queue.cc:260] T 00000000000000000000000000000000 P e5c3f1dad6bb496bac7393d86f82278f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e5c3f1dad6bb496bac7393d86f82278f" member_type: VOTER last_known_addr { host: "127.11.102.62" port: 45915 } }
I20260620 06:07:40.998174 20954 raft_consensus.cc:388] T 00000000000000000000000000000000 P e5c3f1dad6bb496bac7393d86f82278f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 06:07:40.998216 20954 raft_consensus.cc:482] T 00000000000000000000000000000000 P e5c3f1dad6bb496bac7393d86f82278f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 06:07:40.998282 20954 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e5c3f1dad6bb496bac7393d86f82278f [term 0 FOLLOWER]: Advancing to term 1
I20260620 06:07:40.999024 20954 raft_consensus.cc:504] T 00000000000000000000000000000000 P e5c3f1dad6bb496bac7393d86f82278f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e5c3f1dad6bb496bac7393d86f82278f" member_type: VOTER last_known_addr { host: "127.11.102.62" port: 45915 } }
I20260620 06:07:40.999300 20954 leader_election.cc:302] T 00000000000000000000000000000000 P e5c3f1dad6bb496bac7393d86f82278f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e5c3f1dad6bb496bac7393d86f82278f; no voters:
I20260620 06:07:40.999528 20954 leader_election.cc:288] T 00000000000000000000000000000000 P e5c3f1dad6bb496bac7393d86f82278f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 06:07:40.999661 20959 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e5c3f1dad6bb496bac7393d86f82278f [term 1 FOLLOWER]: Leader election won for term 1
I20260620 06:07:40.999887 20959 raft_consensus.cc:686] T 00000000000000000000000000000000 P e5c3f1dad6bb496bac7393d86f82278f [term 1 LEADER]: Becoming Leader. State: Replica: e5c3f1dad6bb496bac7393d86f82278f, State: Running, Role: LEADER
I20260620 06:07:41.000228 20959 consensus_queue.cc:237] T 00000000000000000000000000000000 P e5c3f1dad6bb496bac7393d86f82278f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e5c3f1dad6bb496bac7393d86f82278f" member_type: VOTER last_known_addr { host: "127.11.102.62" port: 45915 } }
I20260620 06:07:41.000700 20954 sys_catalog.cc:564] T 00000000000000000000000000000000 P e5c3f1dad6bb496bac7393d86f82278f [sys.catalog]: configured and running, proceeding with master startup.
I20260620 06:07:41.002038 20961 sys_catalog.cc:455] T 00000000000000000000000000000000 P e5c3f1dad6bb496bac7393d86f82278f [sys.catalog]: SysCatalogTable state changed. Reason: New leader e5c3f1dad6bb496bac7393d86f82278f. Latest consensus state: current_term: 1 leader_uuid: "e5c3f1dad6bb496bac7393d86f82278f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e5c3f1dad6bb496bac7393d86f82278f" member_type: VOTER last_known_addr { host: "127.11.102.62" port: 45915 } } }
I20260620 06:07:41.002063 20960 sys_catalog.cc:455] T 00000000000000000000000000000000 P e5c3f1dad6bb496bac7393d86f82278f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e5c3f1dad6bb496bac7393d86f82278f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e5c3f1dad6bb496bac7393d86f82278f" member_type: VOTER last_known_addr { host: "127.11.102.62" port: 45915 } } }
I20260620 06:07:41.002197 20961 sys_catalog.cc:458] T 00000000000000000000000000000000 P e5c3f1dad6bb496bac7393d86f82278f [sys.catalog]: This master's current role is: LEADER
I20260620 06:07:41.002235 20960 sys_catalog.cc:458] T 00000000000000000000000000000000 P e5c3f1dad6bb496bac7393d86f82278f [sys.catalog]: This master's current role is: LEADER
W20260620 06:07:41.005563 20974 catalog_manager.cc:1558] T 00000000000000000000000000000000 P e5c3f1dad6bb496bac7393d86f82278f: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 06:07:41.005640 20974 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 06:07:41.005780 20969 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 06:07:41.006793 20969 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 06:07:41.012634 20969 catalog_manager.cc:1347] Generated new cluster ID: 04b86b9302a3475aaad1fa2b88ee8841
I20260620 06:07:41.012701 20969 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 06:07:41.033511 20969 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 06:07:41.034840 20969 catalog_manager.cc:1504] Loading token signing keys...
I20260620 06:07:41.046314 20969 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e5c3f1dad6bb496bac7393d86f82278f: Generated new TSK 0
I20260620 06:07:41.047029 20969 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 06:07:41.059304 11672 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9HMp43/build/debug/bin/kudu
/tmp/dist-test-task9HMp43/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781935603120653-11672-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781935603120653-11672-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781935603120653-11672-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781935603120653-11672-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.102.1:0
--local_ip_for_outbound_sockets=127.11.102.1
--webserver_interface=127.11.102.1
--webserver_port=0
--tserver_master_addrs=127.11.102.62:45915
--builtin_ntp_servers=127.11.102.20:33047
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260620 06:07:41.165454 20978 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:07:41.165684 20978 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:07:41.165719 20978 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:07:41.169657 20978 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:07:41.169703 20978 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:07:41.169778 20978 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.102.1
I20260620 06:07:41.174283 20978 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.102.20:33047
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781935603120653-11672-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781935603120653-11672-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.102.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781935603120653-11672-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.11.102.1
--webserver_port=0
--tserver_master_addrs=127.11.102.62:45915
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20978
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.102.1
--log_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781935603120653-11672-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 05:28:48 UTC on ccycloud.quasar-fixhms.root.comops.site
build id 4457020
I20260620 06:07:41.175278 20978 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:07:41.176611 20978 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 06:07:41.185777 20986 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:07:41.185791 20984 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:07:41.185773 20983 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:07:41.186056 20978 server_base.cc:1034] running on GCE node
I20260620 06:07:41.186579 20978 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:07:41.187290 20978 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:07:41.188452 20978 hybrid_clock.cc:648] HybridClock initialized: now 1781935661188420 us; error 43 us; skew 500 ppm
I20260620 06:07:41.190115 20978 webserver.cc:460] Webserver started at http://127.11.102.1:44149/ using document root <none> and password file <none>
I20260620 06:07:41.190677 20978 fs_manager.cc:359] Metadata directory not provided
I20260620 06:07:41.190737 20978 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:07:41.190958 20978 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 06:07:41.193038 20978 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781935603120653-11672-0/minicluster-data/ts-0/data/instance:
uuid: "23f5c02559484bdd860602409ed7167c"
format_stamp: "Formatted at 2026-06-20 06:07:41 on dist-test-slave-9hm5"
server_key: "50db0b086ac0d5324caf5772d413759c"
server_key_iv: "ee8e8bda2cc6a7189a8154397fd1bea3"
server_key_version: "encryptionkey@0"
I20260620 06:07:41.193565 20978 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781935603120653-11672-0/minicluster-data/ts-0/wal/instance:
uuid: "23f5c02559484bdd860602409ed7167c"
format_stamp: "Formatted at 2026-06-20 06:07:41 on dist-test-slave-9hm5"
server_key: "50db0b086ac0d5324caf5772d413759c"
server_key_iv: "ee8e8bda2cc6a7189a8154397fd1bea3"
server_key_version: "encryptionkey@0"
I20260620 06:07:41.197055 20978 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260620 06:07:41.199335 20992 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:07:41.200591 20978 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 06:07:41.200699 20978 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781935603120653-11672-0/minicluster-data/ts-0/data,/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781935603120653-11672-0/minicluster-data/ts-0/wal
uuid: "23f5c02559484bdd860602409ed7167c"
format_stamp: "Formatted at 2026-06-20 06:07:41 on dist-test-slave-9hm5"
server_key: "50db0b086ac0d5324caf5772d413759c"
server_key_iv: "ee8e8bda2cc6a7189a8154397fd1bea3"
server_key_version: "encryptionkey@0"
I20260620 06:07:41.200809 20978 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781935603120653-11672-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781935603120653-11672-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781935603120653-11672-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:07:41.211704 20978 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:07:41.212291 20978 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:07:41.212468 20978 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:07:41.212976 20978 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 06:07:41.213862 20978 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 06:07:41.213924 20978 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 06:07:41.214003 20978 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 06:07:41.214057 20978 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 06:07:41.222395 20978 rpc_server.cc:307] RPC server started. Bound to: 127.11.102.1:34037
I20260620 06:07:41.222425 21105 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.102.1:34037 every 8 connection(s)
I20260620 06:07:41.223214 20978 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781935603120653-11672-0/minicluster-data/ts-0/data/info.pb
I20260620 06:07:41.225459 11672 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9HMp43/build/debug/bin/kudu as pid 20978
I20260620 06:07:41.225597 11672 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781935603120653-11672-0/minicluster-data/ts-0/wal/instance
I20260620 06:07:41.225899 11672 external_mini_cluster.cc:1428] Setting key 7af1212240eaff1866857d58fe395fb6
I20260620 06:07:41.227759 11672 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9HMp43/build/debug/bin/kudu
/tmp/dist-test-task9HMp43/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781935603120653-11672-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781935603120653-11672-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781935603120653-11672-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781935603120653-11672-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.102.2:0
--local_ip_for_outbound_sockets=127.11.102.2
--webserver_interface=127.11.102.2
--webserver_port=0
--tserver_master_addrs=127.11.102.62:45915
--builtin_ntp_servers=127.11.102.20:33047
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260620 06:07:41.233018 20957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:07:41.224975 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.102.1:37253 (local address 127.11.102.62:45915)
0620 06:07:41.225113 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0620 06:07:41.225117 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 06:07:41.226177 (+ 1060us) server_negotiation.cc:407] Connection header received
0620 06:07:41.227490 (+ 1313us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:07:41.227497 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:07:41.227559 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:07:41.227686 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:07:41.228528 (+ 842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:07:41.229294 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:07:41.230741 (+ 1447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:07:41.231166 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:07:41.231292 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:07:41.232433 (+ 1141us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:07:41.232438 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:07:41.232444 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 06:07:41.232460 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:07:41.232487 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 06:07:41.232501 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:07:41.232590 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0620 06:07:41.232724 (+ 134us) server_negotiation.cc:299] Negotiation successful
0620 06:07:41.232813 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260620 06:07:41.233777 21108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:07:41.225348 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.102.62:45915 (local address 127.11.102.1:37253)
0620 06:07:41.226013 (+ 665us) negotiation.cc:107] Waiting for socket to connect
0620 06:07:41.226063 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0620 06:07:41.227298 (+ 1235us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:07:41.227831 (+ 533us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:07:41.227837 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:07:41.227916 (+ 79us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:07:41.228390 (+ 474us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:07:41.228402 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:07:41.229446 (+ 1044us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:07:41.229453 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:07:41.230612 (+ 1159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:07:41.230622 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:07:41.231299 (+ 677us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:07:41.231301 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:07:41.231509 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:07:41.232186 (+ 677us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 06:07:41.232187 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:07:41.232223 (+ 36us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 06:07:41.232278 (+ 55us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:07:41.232588 (+ 310us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:07:41.232594 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:07:41.232597 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 06:07:41.232688 (+ 91us) client_negotiation.cc:240] Negotiation successful
0620 06:07:41.232887 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":246,"thread_start_us":126,"threads_started":1}
I20260620 06:07:41.234908 21106 heartbeater.cc:347] Connected to a master server at 127.11.102.62:45915
I20260620 06:07:41.235152 21106 heartbeater.cc:464] Registering TS with master...
I20260620 06:07:41.235773 21106 heartbeater.cc:511] Master 127.11.102.62:45915 requested a full tablet report, sending...
I20260620 06:07:41.237329 20912 ts_manager.cc:194] Registered new tserver with Master: 23f5c02559484bdd860602409ed7167c (127.11.102.1:34037)
I20260620 06:07:41.238443 20912 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.11.102.1:37253
W20260620 06:07:41.337391 21109 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:07:41.337648 21109 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:07:41.337685 21109 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:07:41.341681 21109 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:07:41.341735 21109 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:07:41.341789 21109 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.102.2
I20260620 06:07:41.346544 21109 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.102.20:33047
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781935603120653-11672-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781935603120653-11672-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.102.2:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781935603120653-11672-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.11.102.2
--webserver_port=0
--tserver_master_addrs=127.11.102.62:45915
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21109
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.102.2
--log_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781935603120653-11672-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 05:28:48 UTC on ccycloud.quasar-fixhms.root.comops.site
build id 4457020
I20260620 06:07:41.347565 21109 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:07:41.348961 21109 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 06:07:41.358284 21109 server_base.cc:1034] running on GCE node
W20260620 06:07:41.358260 21117 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:07:41.358280 21115 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:07:41.358408 21114 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:07:41.358769 21109 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:07:41.359375 21109 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:07:41.360553 21109 hybrid_clock.cc:648] HybridClock initialized: now 1781935661360522 us; error 44 us; skew 500 ppm
I20260620 06:07:41.362385 21109 webserver.cc:460] Webserver started at http://127.11.102.2:35389/ using document root <none> and password file <none>
I20260620 06:07:41.362999 21109 fs_manager.cc:359] Metadata directory not provided
I20260620 06:07:41.363085 21109 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:07:41.363355 21109 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 06:07:41.365343 21109 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781935603120653-11672-0/minicluster-data/ts-1/data/instance:
uuid: "98183ba997924a7f84226878346609cd"
format_stamp: "Formatted at 2026-06-20 06:07:41 on dist-test-slave-9hm5"
server_key: "14c7add89478719697ec056901fb3077"
server_key_iv: "5795f2f16cf83ce9a2a6d6af7b44379d"
server_key_version: "encryptionkey@0"
I20260620 06:07:41.365890 21109 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781935603120653-11672-0/minicluster-data/ts-1/wal/instance:
uuid: "98183ba997924a7f84226878346609cd"
format_stamp: "Formatted at 2026-06-20 06:07:41 on dist-test-slave-9hm5"
server_key: "14c7add89478719697ec056901fb3077"
server_key_iv: "5795f2f16cf83ce9a2a6d6af7b44379d"
server_key_version: "encryptionkey@0"
I20260620 06:07:41.369781 21109 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260620 06:07:41.372148 21123 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:07:41.373296 21109 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 06:07:41.373421 21109 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781935603120653-11672-0/minicluster-data/ts-1/data,/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781935603120653-11672-0/minicluster-data/ts-1/wal
uuid: "98183ba997924a7f84226878346609cd"
format_stamp: "Formatted at 2026-06-20 06:07:41 on dist-test-slave-9hm5"
server_key: "14c7add89478719697ec056901fb3077"
server_key_iv: "5795f2f16cf83ce9a2a6d6af7b44379d"
server_key_version: "encryptionkey@0"
I20260620 06:07:41.373525 21109 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781935603120653-11672-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781935603120653-11672-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781935603120653-11672-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:07:41.385222 21109 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:07:41.385824 21109 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:07:41.386003 21109 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:07:41.386516 21109 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 06:07:41.387372 21109 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 06:07:41.387440 21109 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 06:07:41.387512 21109 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 06:07:41.387553 21109 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 06:07:41.395792 21109 rpc_server.cc:307] RPC server started. Bound to: 127.11.102.2:45329
I20260620 06:07:41.395840 21236 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.102.2:45329 every 8 connection(s)
I20260620 06:07:41.396653 21109 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781935603120653-11672-0/minicluster-data/ts-1/data/info.pb
I20260620 06:07:41.403971 11672 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9HMp43/build/debug/bin/kudu as pid 21109
I20260620 06:07:41.404096 11672 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781935603120653-11672-0/minicluster-data/ts-1/wal/instance
I20260620 06:07:41.404397 11672 external_mini_cluster.cc:1428] Setting key 3eed87f2be525bbcbdc62f432bd11a5d
I20260620 06:07:41.406147 20957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:07:41.398375 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.102.2:53537 (local address 127.11.102.62:45915)
0620 06:07:41.398524 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0620 06:07:41.398528 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 06:07:41.399496 (+ 968us) server_negotiation.cc:407] Connection header received
0620 06:07:41.400966 (+ 1470us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:07:41.400970 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:07:41.401022 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:07:41.401122 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:07:41.402121 (+ 999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:07:41.402709 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:07:41.403960 (+ 1251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:07:41.404267 (+ 307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:07:41.404372 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:07:41.405623 (+ 1251us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:07:41.405627 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:07:41.405629 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 06:07:41.405639 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:07:41.405674 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 06:07:41.405697 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:07:41.405774 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0620 06:07:41.405958 (+ 184us) server_negotiation.cc:299] Negotiation successful
0620 06:07:41.406016 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260620 06:07:41.406560 11672 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9HMp43/build/debug/bin/kudu
/tmp/dist-test-task9HMp43/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781935603120653-11672-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781935603120653-11672-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781935603120653-11672-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781935603120653-11672-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.102.3:0
--local_ip_for_outbound_sockets=127.11.102.3
--webserver_interface=127.11.102.3
--webserver_port=0
--tserver_master_addrs=127.11.102.62:45915
--builtin_ntp_servers=127.11.102.20:33047
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260620 06:07:41.407013 21239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:07:41.398744 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.102.62:45915 (local address 127.11.102.2:53537)
0620 06:07:41.399353 (+ 609us) negotiation.cc:107] Waiting for socket to connect
0620 06:07:41.399390 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0620 06:07:41.400715 (+ 1325us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:07:41.401302 (+ 587us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:07:41.401309 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:07:41.401431 (+ 122us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:07:41.401985 (+ 554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:07:41.402002 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:07:41.402825 (+ 823us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:07:41.402829 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:07:41.403846 (+ 1017us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:07:41.403855 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:07:41.404406 (+ 551us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:07:41.404410 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:07:41.404649 (+ 239us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:07:41.405367 (+ 718us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 06:07:41.405368 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:07:41.405416 (+ 48us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 06:07:41.405485 (+ 69us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:07:41.405788 (+ 303us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:07:41.405798 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:07:41.405807 (+ 9us) client_negotiation.cc:769] Sending connection context
0620 06:07:41.405935 (+ 128us) client_negotiation.cc:240] Negotiation successful
0620 06:07:41.406161 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":237,"thread_start_us":107,"threads_started":1}
I20260620 06:07:41.408094 21237 heartbeater.cc:347] Connected to a master server at 127.11.102.62:45915
I20260620 06:07:41.408324 21237 heartbeater.cc:464] Registering TS with master...
I20260620 06:07:41.408783 21237 heartbeater.cc:511] Master 127.11.102.62:45915 requested a full tablet report, sending...
I20260620 06:07:41.409706 20912 ts_manager.cc:194] Registered new tserver with Master: 98183ba997924a7f84226878346609cd (127.11.102.2:45329)
I20260620 06:07:41.410267 20912 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.11.102.2:53537
W20260620 06:07:41.514736 21240 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:07:41.514988 21240 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:07:41.515023 21240 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:07:41.519023 21240 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:07:41.519070 21240 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:07:41.519152 21240 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.102.3
I20260620 06:07:41.523819 21240 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.102.20:33047
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781935603120653-11672-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781935603120653-11672-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.102.3:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781935603120653-11672-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.11.102.3
--webserver_port=0
--tserver_master_addrs=127.11.102.62:45915
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21240
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.102.3
--log_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781935603120653-11672-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 05:28:48 UTC on ccycloud.quasar-fixhms.root.comops.site
build id 4457020
I20260620 06:07:41.524807 21240 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:07:41.526140 21240 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 06:07:41.535199 21245 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:07:41.535225 21248 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:07:41.535293 21240 server_base.cc:1034] running on GCE node
W20260620 06:07:41.535245 21246 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:07:41.535872 21240 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:07:41.536444 21240 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:07:41.537593 21240 hybrid_clock.cc:648] HybridClock initialized: now 1781935661537562 us; error 45 us; skew 500 ppm
I20260620 06:07:41.539321 21240 webserver.cc:460] Webserver started at http://127.11.102.3:41409/ using document root <none> and password file <none>
I20260620 06:07:41.539918 21240 fs_manager.cc:359] Metadata directory not provided
I20260620 06:07:41.539979 21240 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:07:41.540174 21240 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 06:07:41.542153 21240 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781935603120653-11672-0/minicluster-data/ts-2/data/instance:
uuid: "6193d05e4e9546c38b1e447433ea53c6"
format_stamp: "Formatted at 2026-06-20 06:07:41 on dist-test-slave-9hm5"
server_key: "bb6b24ff004cb013c4cd6843c37d71ff"
server_key_iv: "fbea7218a987ccc508434b89c9018bbb"
server_key_version: "encryptionkey@0"
I20260620 06:07:41.542691 21240 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781935603120653-11672-0/minicluster-data/ts-2/wal/instance:
uuid: "6193d05e4e9546c38b1e447433ea53c6"
format_stamp: "Formatted at 2026-06-20 06:07:41 on dist-test-slave-9hm5"
server_key: "bb6b24ff004cb013c4cd6843c37d71ff"
server_key_iv: "fbea7218a987ccc508434b89c9018bbb"
server_key_version: "encryptionkey@0"
I20260620 06:07:41.546355 21240 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260620 06:07:41.548777 21254 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:07:41.549944 21240 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260620 06:07:41.550112 21240 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781935603120653-11672-0/minicluster-data/ts-2/data,/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781935603120653-11672-0/minicluster-data/ts-2/wal
uuid: "6193d05e4e9546c38b1e447433ea53c6"
format_stamp: "Formatted at 2026-06-20 06:07:41 on dist-test-slave-9hm5"
server_key: "bb6b24ff004cb013c4cd6843c37d71ff"
server_key_iv: "fbea7218a987ccc508434b89c9018bbb"
server_key_version: "encryptionkey@0"
I20260620 06:07:41.550220 21240 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781935603120653-11672-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781935603120653-11672-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781935603120653-11672-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:07:41.573532 21240 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:07:41.574447 21240 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:07:41.574681 21240 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:07:41.575318 21240 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 06:07:41.576371 21240 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 06:07:41.576421 21240 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 06:07:41.576459 21240 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 06:07:41.576515 21240 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 06:07:41.586012 21240 rpc_server.cc:307] RPC server started. Bound to: 127.11.102.3:40157
I20260620 06:07:41.586032 21367 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.102.3:40157 every 8 connection(s)
I20260620 06:07:41.586934 21240 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781935603120653-11672-0/minicluster-data/ts-2/data/info.pb
I20260620 06:07:41.592820 11672 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9HMp43/build/debug/bin/kudu as pid 21240
I20260620 06:07:41.592926 11672 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781935603120653-11672-0/minicluster-data/ts-2/wal/instance
I20260620 06:07:41.593212 11672 external_mini_cluster.cc:1428] Setting key 91410ed52a669a39eee74269e9575bd5
I20260620 06:07:41.596884 20957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:07:41.588730 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.102.3:42053 (local address 127.11.102.62:45915)
0620 06:07:41.588874 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0620 06:07:41.588879 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 06:07:41.589938 (+ 1059us) server_negotiation.cc:407] Connection header received
0620 06:07:41.591440 (+ 1502us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:07:41.591443 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:07:41.591501 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:07:41.591583 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:07:41.592465 (+ 882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:07:41.593296 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:07:41.594987 (+ 1691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:07:41.595332 (+ 345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:07:41.595442 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:07:41.596473 (+ 1031us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:07:41.596476 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:07:41.596478 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 06:07:41.596489 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:07:41.596508 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 06:07:41.596516 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:07:41.596581 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0620 06:07:41.596721 (+ 140us) server_negotiation.cc:299] Negotiation successful
0620 06:07:41.596771 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260620 06:07:41.597707 21370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:07:41.589078 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.102.62:45915 (local address 127.11.102.3:42053)
0620 06:07:41.589741 (+ 663us) negotiation.cc:107] Waiting for socket to connect
0620 06:07:41.589782 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0620 06:07:41.591213 (+ 1431us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:07:41.591769 (+ 556us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:07:41.591776 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:07:41.591863 (+ 87us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:07:41.592326 (+ 463us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:07:41.592337 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:07:41.593429 (+ 1092us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:07:41.593432 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:07:41.594854 (+ 1422us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:07:41.594865 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:07:41.595482 (+ 617us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:07:41.595484 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:07:41.595680 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:07:41.596224 (+ 544us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 06:07:41.596225 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:07:41.596263 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 06:07:41.596322 (+ 59us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:07:41.596593 (+ 271us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:07:41.596599 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:07:41.596604 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 06:07:41.596694 (+ 90us) client_negotiation.cc:240] Negotiation successful
0620 06:07:41.596902 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":270,"thread_start_us":111,"threads_started":1}
I20260620 06:07:41.598795 21368 heartbeater.cc:347] Connected to a master server at 127.11.102.62:45915
I20260620 06:07:41.599020 21368 heartbeater.cc:464] Registering TS with master...
I20260620 06:07:41.599517 21368 heartbeater.cc:511] Master 127.11.102.62:45915 requested a full tablet report, sending...
I20260620 06:07:41.600560 20912 ts_manager.cc:194] Registered new tserver with Master: 6193d05e4e9546c38b1e447433ea53c6 (127.11.102.3:40157)
I20260620 06:07:41.601135 20912 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.11.102.3:42053
I20260620 06:07:41.607185 11672 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260620 06:07:41.610011 20957 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 06:07:41.609046 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40188 (local address 127.11.102.62:45915)
0620 06:07:41.609201 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0620 06:07:41.609206 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 06:07:41.609390 (+ 184us) server_negotiation.cc:407] Connection header received
0620 06:07:41.609539 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:07:41.609542 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:07:41.609585 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:07:41.609675 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:07:41.609900 (+ 225us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40188: BlockingRecv error: recv got EOF from 127.0.0.1:40188 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
I20260620 06:07:41.610146 11672 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9HMp43/build/debug/bin/kudu with pid 20978
I20260620 06:07:41.615319 11672 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9HMp43/build/debug/bin/kudu with pid 21109
I20260620 06:07:41.620692 11672 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9HMp43/build/debug/bin/kudu with pid 21240
I20260620 06:07:41.625883 11672 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9HMp43/build/debug/bin/kudu with pid 20887
2026-06-20T06:07:41Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (840 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-06-20T06:07:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T06:07:41Z Disabled control of system clock
I20260620 06:07:41.661054 11672 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9HMp43/build/debug/bin/kudu
/tmp/dist-test-task9HMp43/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781935603120653-11672-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781935603120653-11672-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781935603120653-11672-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781935603120653-11672-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.102.62:45909
--webserver_interface=127.11.102.62
--webserver_port=0
--builtin_ntp_servers=127.11.102.20:39105
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.102.62:45909
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260620 06:07:41.769327 21379 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:07:41.769583 21379 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:07:41.769618 21379 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:07:41.773576 21379 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 06:07:41.773633 21379 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:07:41.773649 21379 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:07:41.773667 21379 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 06:07:41.773682 21379 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 06:07:41.778501 21379 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.102.20:39105
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781935603120653-11672-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781935603120653-11672-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.102.62:45909
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.102.62:45909
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781935603120653-11672-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.102.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21379
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781935603120653-11672-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 05:28:48 UTC on ccycloud.quasar-fixhms.root.comops.site
build id 4457020
I20260620 06:07:41.779531 21379 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:07:41.780934 21379 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 06:07:41.789113 21384 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:07:41.789165 21385 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:07:41.789222 21379 server_base.cc:1034] running on GCE node
W20260620 06:07:41.789175 21387 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:07:41.789912 21379 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:07:41.790884 21379 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:07:41.792068 21379 hybrid_clock.cc:648] HybridClock initialized: now 1781935661792054 us; error 32 us; skew 500 ppm
I20260620 06:07:41.793857 21379 webserver.cc:460] Webserver started at http://127.11.102.62:36019/ using document root <none> and password file <none>
I20260620 06:07:41.794441 21379 fs_manager.cc:359] Metadata directory not provided
I20260620 06:07:41.794533 21379 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:07:41.794752 21379 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 06:07:41.796787 21379 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781935603120653-11672-0/minicluster-data/master-0/data/instance:
uuid: "459c0e7713ac485c80497c29bf08c633"
format_stamp: "Formatted at 2026-06-20 06:07:41 on dist-test-slave-9hm5"
server_key: "90b05aa797572efd80f45f265e08112f"
server_key_iv: "e046eb4cc910047f766d9b1156f0edef"
server_key_version: "encryptionkey@0"
I20260620 06:07:41.797338 21379 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781935603120653-11672-0/minicluster-data/master-0/wal/instance:
uuid: "459c0e7713ac485c80497c29bf08c633"
format_stamp: "Formatted at 2026-06-20 06:07:41 on dist-test-slave-9hm5"
server_key: "90b05aa797572efd80f45f265e08112f"
server_key_iv: "e046eb4cc910047f766d9b1156f0edef"
server_key_version: "encryptionkey@0"
I20260620 06:07:41.800877 21379 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 06:07:41.803236 21393 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:07:41.804464 21379 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 06:07:41.804600 21379 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781935603120653-11672-0/minicluster-data/master-0/data,/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781935603120653-11672-0/minicluster-data/master-0/wal
uuid: "459c0e7713ac485c80497c29bf08c633"
format_stamp: "Formatted at 2026-06-20 06:07:41 on dist-test-slave-9hm5"
server_key: "90b05aa797572efd80f45f265e08112f"
server_key_iv: "e046eb4cc910047f766d9b1156f0edef"
server_key_version: "encryptionkey@0"
I20260620 06:07:41.804709 21379 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781935603120653-11672-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781935603120653-11672-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781935603120653-11672-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:07:41.809028 21379 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:07:41.809602 21379 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:07:41.809779 21379 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:07:41.816073 21379 rpc_server.cc:307] RPC server started. Bound to: 127.11.102.62:45909
I20260620 06:07:41.816241 21445 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.102.62:45909 every 8 connection(s)
I20260620 06:07:41.817008 21379 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781935603120653-11672-0/minicluster-data/master-0/data/info.pb
I20260620 06:07:41.820113 21446 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 06:07:41.825212 21446 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 459c0e7713ac485c80497c29bf08c633: Bootstrap starting.
I20260620 06:07:41.826527 11672 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9HMp43/build/debug/bin/kudu as pid 21379
I20260620 06:07:41.826646 11672 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781935603120653-11672-0/minicluster-data/master-0/wal/instance
I20260620 06:07:41.826922 11672 external_mini_cluster.cc:1428] Setting key ba9a708dbd7d04d7aade750c74223b05
I20260620 06:07:41.827891 21446 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 459c0e7713ac485c80497c29bf08c633: Neither blocks nor log segments found. Creating new log.
I20260620 06:07:41.828681 21446 log.cc:824] T 00000000000000000000000000000000 P 459c0e7713ac485c80497c29bf08c633: Log is configured to *not* fsync() on all Append() calls
I20260620 06:07:41.830533 21446 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 459c0e7713ac485c80497c29bf08c633: No bootstrap required, opened a new log
I20260620 06:07:41.832074 21449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:07:41.828042 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36150 (local address 127.11.102.62:45909)
0620 06:07:41.828697 (+ 655us) server_negotiation.cc:206] Beginning negotiation
0620 06:07:41.828717 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0620 06:07:41.828788 (+ 71us) server_negotiation.cc:407] Connection header received
0620 06:07:41.829406 (+ 618us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:07:41.829422 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:07:41.829707 (+ 285us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:07:41.830039 (+ 332us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:07:41.830826 (+ 787us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:07:41.830830 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:07:41.830842 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 06:07:41.830851 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:07:41.830902 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 06:07:41.830917 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:07:41.831034 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0620 06:07:41.831150 (+ 116us) server_negotiation.cc:299] Negotiation successful
0620 06:07:41.831371 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":297,"thread_start_us":121,"threads_started":1}
I20260620 06:07:41.834084 21446 raft_consensus.cc:348] T 00000000000000000000000000000000 P 459c0e7713ac485c80497c29bf08c633 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "459c0e7713ac485c80497c29bf08c633" member_type: VOTER last_known_addr { host: "127.11.102.62" port: 45909 } }
I20260620 06:07:41.834306 21446 raft_consensus.cc:374] T 00000000000000000000000000000000 P 459c0e7713ac485c80497c29bf08c633 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 06:07:41.834383 21446 raft_consensus.cc:729] T 00000000000000000000000000000000 P 459c0e7713ac485c80497c29bf08c633 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 459c0e7713ac485c80497c29bf08c633, State: Initialized, Role: FOLLOWER
I20260620 06:07:41.834839 21446 consensus_queue.cc:260] T 00000000000000000000000000000000 P 459c0e7713ac485c80497c29bf08c633 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "459c0e7713ac485c80497c29bf08c633" member_type: VOTER last_known_addr { host: "127.11.102.62" port: 45909 } }
I20260620 06:07:41.834980 21446 raft_consensus.cc:388] T 00000000000000000000000000000000 P 459c0e7713ac485c80497c29bf08c633 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 06:07:41.835071 21446 raft_consensus.cc:482] T 00000000000000000000000000000000 P 459c0e7713ac485c80497c29bf08c633 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 06:07:41.835178 21446 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 459c0e7713ac485c80497c29bf08c633 [term 0 FOLLOWER]: Advancing to term 1
I20260620 06:07:41.836092 21446 raft_consensus.cc:504] T 00000000000000000000000000000000 P 459c0e7713ac485c80497c29bf08c633 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "459c0e7713ac485c80497c29bf08c633" member_type: VOTER last_known_addr { host: "127.11.102.62" port: 45909 } }
I20260620 06:07:41.836431 21446 leader_election.cc:302] T 00000000000000000000000000000000 P 459c0e7713ac485c80497c29bf08c633 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 459c0e7713ac485c80497c29bf08c633; no voters:
I20260620 06:07:41.836781 21446 leader_election.cc:288] T 00000000000000000000000000000000 P 459c0e7713ac485c80497c29bf08c633 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 06:07:41.836880 21451 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 459c0e7713ac485c80497c29bf08c633 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 06:07:41.837044 21451 raft_consensus.cc:686] T 00000000000000000000000000000000 P 459c0e7713ac485c80497c29bf08c633 [term 1 LEADER]: Becoming Leader. State: Replica: 459c0e7713ac485c80497c29bf08c633, State: Running, Role: LEADER
I20260620 06:07:41.837309 21451 consensus_queue.cc:237] T 00000000000000000000000000000000 P 459c0e7713ac485c80497c29bf08c633 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "459c0e7713ac485c80497c29bf08c633" member_type: VOTER last_known_addr { host: "127.11.102.62" port: 45909 } }
I20260620 06:07:41.837944 21446 sys_catalog.cc:564] T 00000000000000000000000000000000 P 459c0e7713ac485c80497c29bf08c633 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 06:07:41.838980 21453 sys_catalog.cc:455] T 00000000000000000000000000000000 P 459c0e7713ac485c80497c29bf08c633 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "459c0e7713ac485c80497c29bf08c633" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "459c0e7713ac485c80497c29bf08c633" member_type: VOTER last_known_addr { host: "127.11.102.62" port: 45909 } } }
I20260620 06:07:41.839002 21452 sys_catalog.cc:455] T 00000000000000000000000000000000 P 459c0e7713ac485c80497c29bf08c633 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 459c0e7713ac485c80497c29bf08c633. Latest consensus state: current_term: 1 leader_uuid: "459c0e7713ac485c80497c29bf08c633" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "459c0e7713ac485c80497c29bf08c633" member_type: VOTER last_known_addr { host: "127.11.102.62" port: 45909 } } }
I20260620 06:07:41.839104 21452 sys_catalog.cc:458] T 00000000000000000000000000000000 P 459c0e7713ac485c80497c29bf08c633 [sys.catalog]: This master's current role is: LEADER
I20260620 06:07:41.839103 21453 sys_catalog.cc:458] T 00000000000000000000000000000000 P 459c0e7713ac485c80497c29bf08c633 [sys.catalog]: This master's current role is: LEADER
I20260620 06:07:41.839466 21460 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 06:07:41.842526 21460 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 06:07:41.847467 21460 catalog_manager.cc:1347] Generated new cluster ID: 42331833ea0542dc991a2ade33b75a7c
I20260620 06:07:41.847548 21460 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 06:07:41.865584 21460 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 06:07:41.865901 21460 catalog_manager.cc:1504] Loading token signing keys...
I20260620 06:07:41.874223 21460 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 459c0e7713ac485c80497c29bf08c633: Generated new TSK 0
I20260620 06:07:41.874894 21460 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 06:07:41.879411 11672 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9HMp43/build/debug/bin/kudu
/tmp/dist-test-task9HMp43/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781935603120653-11672-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781935603120653-11672-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781935603120653-11672-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781935603120653-11672-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.102.1:0
--local_ip_for_outbound_sockets=127.11.102.1
--webserver_interface=127.11.102.1
--webserver_port=0
--tserver_master_addrs=127.11.102.62:45909
--builtin_ntp_servers=127.11.102.20:39105
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260620 06:07:41.994557 21470 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:07:41.994827 21470 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:07:41.994892 21470 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:07:41.998986 21470 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:07:41.999061 21470 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:07:41.999143 21470 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.102.1
I20260620 06:07:42.003816 21470 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.102.20:39105
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781935603120653-11672-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781935603120653-11672-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.102.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781935603120653-11672-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.11.102.1
--webserver_port=0
--tserver_master_addrs=127.11.102.62:45909
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21470
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.102.1
--log_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781935603120653-11672-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 05:28:48 UTC on ccycloud.quasar-fixhms.root.comops.site
build id 4457020
I20260620 06:07:42.004808 21470 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:07:42.006170 21470 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 06:07:42.015978 21475 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:07:42.015990 21476 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:07:42.016078 21478 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:07:42.016623 21470 server_base.cc:1034] running on GCE node
I20260620 06:07:42.017010 21470 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:07:42.017668 21470 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:07:42.018853 21470 hybrid_clock.cc:648] HybridClock initialized: now 1781935662018827 us; error 33 us; skew 500 ppm
I20260620 06:07:42.020565 21470 webserver.cc:460] Webserver started at http://127.11.102.1:41511/ using document root <none> and password file <none>
I20260620 06:07:42.021104 21470 fs_manager.cc:359] Metadata directory not provided
I20260620 06:07:42.021196 21470 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:07:42.021420 21470 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 06:07:42.023291 21470 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781935603120653-11672-0/minicluster-data/ts-0/data/instance:
uuid: "873e607ef1d14140af5dc166b4b3d5b4"
format_stamp: "Formatted at 2026-06-20 06:07:42 on dist-test-slave-9hm5"
server_key: "e8441755655792f3f3b7128fd363b598"
server_key_iv: "5305153cf2076e12105ba11b4fc03e07"
server_key_version: "encryptionkey@0"
I20260620 06:07:42.023850 21470 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781935603120653-11672-0/minicluster-data/ts-0/wal/instance:
uuid: "873e607ef1d14140af5dc166b4b3d5b4"
format_stamp: "Formatted at 2026-06-20 06:07:42 on dist-test-slave-9hm5"
server_key: "e8441755655792f3f3b7128fd363b598"
server_key_iv: "5305153cf2076e12105ba11b4fc03e07"
server_key_version: "encryptionkey@0"
I20260620 06:07:42.027323 21470 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.001s
I20260620 06:07:42.029912 21484 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:07:42.031098 21470 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260620 06:07:42.031234 21470 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781935603120653-11672-0/minicluster-data/ts-0/data,/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781935603120653-11672-0/minicluster-data/ts-0/wal
uuid: "873e607ef1d14140af5dc166b4b3d5b4"
format_stamp: "Formatted at 2026-06-20 06:07:42 on dist-test-slave-9hm5"
server_key: "e8441755655792f3f3b7128fd363b598"
server_key_iv: "5305153cf2076e12105ba11b4fc03e07"
server_key_version: "encryptionkey@0"
I20260620 06:07:42.031348 21470 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781935603120653-11672-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781935603120653-11672-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781935603120653-11672-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:07:42.035852 21470 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:07:42.036494 21470 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:07:42.036697 21470 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:07:42.037221 21470 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 06:07:42.038093 21470 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 06:07:42.038164 21470 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 06:07:42.038242 21470 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 06:07:42.038367 21470 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 06:07:42.046751 21470 rpc_server.cc:307] RPC server started. Bound to: 127.11.102.1:39641
I20260620 06:07:42.046818 21597 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.102.1:39641 every 8 connection(s)
I20260620 06:07:42.047645 21470 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781935603120653-11672-0/minicluster-data/ts-0/data/info.pb
I20260620 06:07:42.053893 21449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:07:42.049417 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.102.1:50081 (local address 127.11.102.62:45909)
0620 06:07:42.049544 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0620 06:07:42.049547 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 06:07:42.050758 (+ 1211us) server_negotiation.cc:407] Connection header received
0620 06:07:42.052333 (+ 1575us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:07:42.052337 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:07:42.052377 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:07:42.052462 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:07:42.053526 (+ 1064us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:07:42.053529 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:07:42.053532 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 06:07:42.053543 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:07:42.053561 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 06:07:42.053569 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:07:42.053646 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0620 06:07:42.053758 (+ 112us) server_negotiation.cc:299] Negotiation successful
0620 06:07:42.053794 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260620 06:07:42.054674 21600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:07:42.049786 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.102.62:45909 (local address 127.11.102.1:50081)
0620 06:07:42.050610 (+ 824us) negotiation.cc:107] Waiting for socket to connect
0620 06:07:42.050651 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0620 06:07:42.051973 (+ 1322us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:07:42.052608 (+ 635us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:07:42.052614 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:07:42.052695 (+ 81us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:07:42.053282 (+ 587us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 06:07:42.053283 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:07:42.053321 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 06:07:42.053392 (+ 71us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:07:42.053645 (+ 253us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:07:42.053651 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:07:42.053654 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 06:07:42.053776 (+ 122us) client_negotiation.cc:240] Negotiation successful
0620 06:07:42.053950 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":292,"thread_start_us":144,"threads_started":1}
I20260620 06:07:42.055838 21598 heartbeater.cc:347] Connected to a master server at 127.11.102.62:45909
I20260620 06:07:42.056066 21598 heartbeater.cc:464] Registering TS with master...
I20260620 06:07:42.056147 11672 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9HMp43/build/debug/bin/kudu as pid 21470
I20260620 06:07:42.056258 11672 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781935603120653-11672-0/minicluster-data/ts-0/wal/instance
I20260620 06:07:42.056516 21598 heartbeater.cc:511] Master 127.11.102.62:45909 requested a full tablet report, sending...
I20260620 06:07:42.056591 11672 external_mini_cluster.cc:1428] Setting key c26e3d7f4f7db8d9d99d38a5f9499fb2
I20260620 06:07:42.058125 21409 ts_manager.cc:194] Registered new tserver with Master: 873e607ef1d14140af5dc166b4b3d5b4 (127.11.102.1:39641)
I20260620 06:07:42.058648 11672 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9HMp43/build/debug/bin/kudu
/tmp/dist-test-task9HMp43/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781935603120653-11672-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781935603120653-11672-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781935603120653-11672-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781935603120653-11672-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.102.2:0
--local_ip_for_outbound_sockets=127.11.102.2
--webserver_interface=127.11.102.2
--webserver_port=0
--tserver_master_addrs=127.11.102.62:45909
--builtin_ntp_servers=127.11.102.20:39105
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260620 06:07:42.165715 21601 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:07:42.165939 21601 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:07:42.165975 21601 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:07:42.169844 21601 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:07:42.169888 21601 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:07:42.169971 21601 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.102.2
I20260620 06:07:42.174538 21601 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.102.20:39105
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781935603120653-11672-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781935603120653-11672-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.102.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781935603120653-11672-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.11.102.2
--webserver_port=0
--tserver_master_addrs=127.11.102.62:45909
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21601
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.102.2
--log_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781935603120653-11672-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 05:28:48 UTC on ccycloud.quasar-fixhms.root.comops.site
build id 4457020
I20260620 06:07:42.175503 21601 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:07:42.176846 21601 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 06:07:42.186067 21607 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:07:42.186139 21601 server_base.cc:1034] running on GCE node
W20260620 06:07:42.186053 21609 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:07:42.186071 21606 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:07:42.186672 21601 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:07:42.187281 21601 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:07:42.188467 21601 hybrid_clock.cc:648] HybridClock initialized: now 1781935662188441 us; error 53 us; skew 500 ppm
I20260620 06:07:42.190162 21601 webserver.cc:460] Webserver started at http://127.11.102.2:36871/ using document root <none> and password file <none>
I20260620 06:07:42.190728 21601 fs_manager.cc:359] Metadata directory not provided
I20260620 06:07:42.190817 21601 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:07:42.191033 21601 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 06:07:42.193044 21601 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781935603120653-11672-0/minicluster-data/ts-1/data/instance:
uuid: "5efe7e83fabe448891c2c2dd5043248d"
format_stamp: "Formatted at 2026-06-20 06:07:42 on dist-test-slave-9hm5"
server_key: "86dbe5cfc5b62f7cfc30f453fdf4ac73"
server_key_iv: "ec06d52206f12951180e6596683a917d"
server_key_version: "encryptionkey@0"
I20260620 06:07:42.193583 21601 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781935603120653-11672-0/minicluster-data/ts-1/wal/instance:
uuid: "5efe7e83fabe448891c2c2dd5043248d"
format_stamp: "Formatted at 2026-06-20 06:07:42 on dist-test-slave-9hm5"
server_key: "86dbe5cfc5b62f7cfc30f453fdf4ac73"
server_key_iv: "ec06d52206f12951180e6596683a917d"
server_key_version: "encryptionkey@0"
I20260620 06:07:42.197192 21601 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260620 06:07:42.199551 21615 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:07:42.200711 21601 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 06:07:42.200843 21601 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781935603120653-11672-0/minicluster-data/ts-1/data,/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781935603120653-11672-0/minicluster-data/ts-1/wal
uuid: "5efe7e83fabe448891c2c2dd5043248d"
format_stamp: "Formatted at 2026-06-20 06:07:42 on dist-test-slave-9hm5"
server_key: "86dbe5cfc5b62f7cfc30f453fdf4ac73"
server_key_iv: "ec06d52206f12951180e6596683a917d"
server_key_version: "encryptionkey@0"
I20260620 06:07:42.200953 21601 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781935603120653-11672-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781935603120653-11672-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781935603120653-11672-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:07:42.205366 21601 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:07:42.205926 21601 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:07:42.206125 21601 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:07:42.206637 21601 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 06:07:42.207481 21601 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 06:07:42.207551 21601 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 06:07:42.207615 21601 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 06:07:42.207683 21601 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 06:07:42.216326 21601 rpc_server.cc:307] RPC server started. Bound to: 127.11.102.2:41619
I20260620 06:07:42.216346 21728 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.102.2:41619 every 8 connection(s)
I20260620 06:07:42.217160 21601 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781935603120653-11672-0/minicluster-data/ts-1/data/info.pb
I20260620 06:07:42.222901 21449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:07:42.218903 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.102.2:60595 (local address 127.11.102.62:45909)
0620 06:07:42.219060 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0620 06:07:42.219065 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 06:07:42.220098 (+ 1033us) server_negotiation.cc:407] Connection header received
0620 06:07:42.221367 (+ 1269us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:07:42.221369 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:07:42.221402 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:07:42.221479 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:07:42.222545 (+ 1066us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:07:42.222548 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:07:42.222550 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 06:07:42.222559 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:07:42.222574 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 06:07:42.222580 (+ 6us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:07:42.222650 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0620 06:07:42.222764 (+ 114us) server_negotiation.cc:299] Negotiation successful
0620 06:07:42.222802 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260620 06:07:42.223784 21731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:07:42.219261 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.102.62:45909 (local address 127.11.102.2:60595)
0620 06:07:42.219963 (+ 702us) negotiation.cc:107] Waiting for socket to connect
0620 06:07:42.220002 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0620 06:07:42.221173 (+ 1171us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:07:42.221626 (+ 453us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:07:42.221632 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:07:42.221718 (+ 86us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:07:42.222315 (+ 597us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 06:07:42.222316 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:07:42.222352 (+ 36us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 06:07:42.222418 (+ 66us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:07:42.222653 (+ 235us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:07:42.222658 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:07:42.222662 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 06:07:42.222802 (+ 140us) client_negotiation.cc:240] Negotiation successful
0620 06:07:42.222975 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":254,"thread_start_us":106,"threads_started":1}
I20260620 06:07:42.224390 11672 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9HMp43/build/debug/bin/kudu as pid 21601
I20260620 06:07:42.224532 11672 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781935603120653-11672-0/minicluster-data/ts-1/wal/instance
I20260620 06:07:42.224838 11672 external_mini_cluster.cc:1428] Setting key acf1cfe5ef9c0556d61ade79d7de8659
I20260620 06:07:42.224897 21729 heartbeater.cc:347] Connected to a master server at 127.11.102.62:45909
I20260620 06:07:42.225099 21729 heartbeater.cc:464] Registering TS with master...
I20260620 06:07:42.225490 21729 heartbeater.cc:511] Master 127.11.102.62:45909 requested a full tablet report, sending...
I20260620 06:07:42.226434 21409 ts_manager.cc:194] Registered new tserver with Master: 5efe7e83fabe448891c2c2dd5043248d (127.11.102.2:41619)
I20260620 06:07:42.226981 11672 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9HMp43/build/debug/bin/kudu
/tmp/dist-test-task9HMp43/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781935603120653-11672-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781935603120653-11672-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781935603120653-11672-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781935603120653-11672-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.102.3:0
--local_ip_for_outbound_sockets=127.11.102.3
--webserver_interface=127.11.102.3
--webserver_port=0
--tserver_master_addrs=127.11.102.62:45909
--builtin_ntp_servers=127.11.102.20:39105
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260620 06:07:42.335179 21732 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:07:42.335443 21732 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:07:42.335482 21732 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:07:42.339983 21732 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:07:42.340037 21732 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:07:42.340119 21732 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.102.3
I20260620 06:07:42.344877 21732 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.102.20:39105
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781935603120653-11672-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781935603120653-11672-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.102.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781935603120653-11672-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.11.102.3
--webserver_port=0
--tserver_master_addrs=127.11.102.62:45909
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21732
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.102.3
--log_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781935603120653-11672-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 05:28:48 UTC on ccycloud.quasar-fixhms.root.comops.site
build id 4457020
I20260620 06:07:42.345867 21732 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:07:42.347179 21732 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 06:07:42.356621 21738 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:07:42.356655 21737 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:07:42.356660 21732 server_base.cc:1034] running on GCE node
W20260620 06:07:42.356629 21740 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:07:42.357247 21732 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:07:42.357860 21732 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:07:42.359045 21732 hybrid_clock.cc:648] HybridClock initialized: now 1781935662359035 us; error 29 us; skew 500 ppm
I20260620 06:07:42.360862 21732 webserver.cc:460] Webserver started at http://127.11.102.3:35103/ using document root <none> and password file <none>
I20260620 06:07:42.361410 21732 fs_manager.cc:359] Metadata directory not provided
I20260620 06:07:42.361500 21732 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:07:42.361728 21732 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 06:07:42.363755 21732 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781935603120653-11672-0/minicluster-data/ts-2/data/instance:
uuid: "b992f3b768844ad6bb11e4d3630bdcb7"
format_stamp: "Formatted at 2026-06-20 06:07:42 on dist-test-slave-9hm5"
server_key: "931f234408e02a0a08c32c7b7f3ea0b5"
server_key_iv: "42b435b93325d0b1c6c8217c108489b9"
server_key_version: "encryptionkey@0"
I20260620 06:07:42.364300 21732 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781935603120653-11672-0/minicluster-data/ts-2/wal/instance:
uuid: "b992f3b768844ad6bb11e4d3630bdcb7"
format_stamp: "Formatted at 2026-06-20 06:07:42 on dist-test-slave-9hm5"
server_key: "931f234408e02a0a08c32c7b7f3ea0b5"
server_key_iv: "42b435b93325d0b1c6c8217c108489b9"
server_key_version: "encryptionkey@0"
I20260620 06:07:42.367802 21732 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 06:07:42.370186 21746 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:07:42.371338 21732 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 06:07:42.371479 21732 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781935603120653-11672-0/minicluster-data/ts-2/data,/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781935603120653-11672-0/minicluster-data/ts-2/wal
uuid: "b992f3b768844ad6bb11e4d3630bdcb7"
format_stamp: "Formatted at 2026-06-20 06:07:42 on dist-test-slave-9hm5"
server_key: "931f234408e02a0a08c32c7b7f3ea0b5"
server_key_iv: "42b435b93325d0b1c6c8217c108489b9"
server_key_version: "encryptionkey@0"
I20260620 06:07:42.371587 21732 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781935603120653-11672-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781935603120653-11672-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781935603120653-11672-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:07:42.376163 21732 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:07:42.376740 21732 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:07:42.376912 21732 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:07:42.377403 21732 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 06:07:42.378242 21732 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 06:07:42.378309 21732 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 06:07:42.378372 21732 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 06:07:42.378422 21732 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 06:07:42.387185 21732 rpc_server.cc:307] RPC server started. Bound to: 127.11.102.3:43153
I20260620 06:07:42.387212 21859 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.102.3:43153 every 8 connection(s)
I20260620 06:07:42.388077 21732 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781935603120653-11672-0/minicluster-data/ts-2/data/info.pb
I20260620 06:07:42.392711 11672 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9HMp43/build/debug/bin/kudu as pid 21732
I20260620 06:07:42.392838 11672 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781935603120653-11672-0/minicluster-data/ts-2/wal/instance
I20260620 06:07:42.393123 11672 external_mini_cluster.cc:1428] Setting key b935096e22ca002022e9065155148a9f
I20260620 06:07:42.394045 21449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:07:42.389786 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.102.3:49897 (local address 127.11.102.62:45909)
0620 06:07:42.389945 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0620 06:07:42.389951 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 06:07:42.390945 (+ 994us) server_negotiation.cc:407] Connection header received
0620 06:07:42.392222 (+ 1277us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:07:42.392225 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:07:42.392265 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:07:42.392350 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:07:42.393502 (+ 1152us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:07:42.393507 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:07:42.393510 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 06:07:42.393526 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:07:42.393555 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 06:07:42.393570 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:07:42.393663 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0620 06:07:42.393845 (+ 182us) server_negotiation.cc:299] Negotiation successful
0620 06:07:42.393890 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260620 06:07:42.394904 21862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:07:42.390186 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.102.62:45909 (local address 127.11.102.3:49897)
0620 06:07:42.390805 (+ 619us) negotiation.cc:107] Waiting for socket to connect
0620 06:07:42.390842 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0620 06:07:42.392028 (+ 1186us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:07:42.392500 (+ 472us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:07:42.392506 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:07:42.392589 (+ 83us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:07:42.393186 (+ 597us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 06:07:42.393187 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:07:42.393254 (+ 67us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 06:07:42.393333 (+ 79us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:07:42.393662 (+ 329us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:07:42.393689 (+ 27us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:07:42.393695 (+ 6us) client_negotiation.cc:769] Sending connection context
0620 06:07:42.393852 (+ 157us) client_negotiation.cc:240] Negotiation successful
0620 06:07:42.394107 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":235,"thread_start_us":115,"threads_started":1}
I20260620 06:07:42.395869 21860 heartbeater.cc:347] Connected to a master server at 127.11.102.62:45909
I20260620 06:07:42.396093 21860 heartbeater.cc:464] Registering TS with master...
I20260620 06:07:42.396497 21860 heartbeater.cc:511] Master 127.11.102.62:45909 requested a full tablet report, sending...
I20260620 06:07:42.397477 21409 ts_manager.cc:194] Registered new tserver with Master: b992f3b768844ad6bb11e4d3630bdcb7 (127.11.102.3:43153)
I20260620 06:07:42.406873 11672 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260620 06:07:42.409866 21449 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 06:07:42.408918 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36152 (local address 127.11.102.62:45909)
0620 06:07:42.409099 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0620 06:07:42.409104 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 06:07:42.409263 (+ 159us) server_negotiation.cc:407] Connection header received
0620 06:07:42.409428 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:07:42.409431 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:07:42.409470 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:07:42.409550 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:07:42.409763 (+ 213us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36152: BlockingRecv error: recv got EOF from 127.0.0.1:36152 (error 108)
Metrics: {"server-negotiator.queue_time_us":91}
I20260620 06:07:42.410184 11672 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9HMp43/build/debug/bin/kudu with pid 21470
I20260620 06:07:42.416100 11672 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9HMp43/build/debug/bin/kudu with pid 21601
I20260620 06:07:42.421885 11672 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9HMp43/build/debug/bin/kudu with pid 21732
I20260620 06:07:42.427166 11672 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9HMp43/build/debug/bin/kudu with pid 21379
2026-06-20T06:07:42Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (798 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781935603120653-11672-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 06:07:42 dist-test-slave-9hm5 krb5kdc[21870](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 06:07:42 dist-test-slave-9hm5 krb5kdc[21870](info): set up 2 sockets
Jun 20 06:07:42 dist-test-slave-9hm5 krb5kdc[21870](info): commencing operation
krb5kdc: starting...
W20260620 06:07:43.464643 11672 mini_kdc.cc:121] Time spent starting KDC: real 1.017s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781935603120653-11672-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781935603120653-11672-0/krb5kdc/test-user.keytab.
Jun 20 06:07:43 dist-test-slave-9hm5 krb5kdc[21870](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935663, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T06:07:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T06:07:43Z Disabled control of system clock
WARNING: no policy specified for kudu/127.11.102.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.102.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.102.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781935603120653-11672-0/krb5kdc/kudu_127.11.102.62.keytab.
Entry for principal kudu/127.11.102.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781935603120653-11672-0/krb5kdc/kudu_127.11.102.62.keytab.
I20260620 06:07:43.584093 11672 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9HMp43/build/debug/bin/kudu
/tmp/dist-test-task9HMp43/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781935603120653-11672-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781935603120653-11672-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781935603120653-11672-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781935603120653-11672-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.102.62:46701
--webserver_interface=127.11.102.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781935603120653-11672-0/krb5kdc/kudu_127.11.102.62.keytab
--principal=kudu/127.11.102.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.102.20:37729
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.102.62:46701
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781935603120653-11672-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781935603120653-11672-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781935603120653-11672-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 06:07:43.692634 21883 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:07:43.692930 21883 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:07:43.693010 21883 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:07:43.696998 21883 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 06:07:43.697084 21883 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:07:43.697124 21883 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:07:43.697182 21883 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 06:07:43.697222 21883 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 06:07:43.702297 21883 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.102.20:37729
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781935603120653-11672-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781935603120653-11672-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.102.62:46701
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781935603120653-11672-0/krb5kdc/kudu_127.11.102.62.keytab
--principal=kudu/127.11.102.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.102.62:46701
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781935603120653-11672-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.102.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21883
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781935603120653-11672-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 05:28:48 UTC on ccycloud.quasar-fixhms.root.comops.site
build id 4457020
I20260620 06:07:43.703294 21883 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:07:43.704276 21883 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 06:07:43.712363 21889 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:07:43.712355 21888 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:07:43.712355 21891 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:07:43.712720 21883 server_base.cc:1034] running on GCE node
I20260620 06:07:43.713256 21883 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:07:43.714238 21883 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:07:43.715416 21883 hybrid_clock.cc:648] HybridClock initialized: now 1781935663715394 us; error 47 us; skew 500 ppm
Jun 20 06:07:43 dist-test-slave-9hm5 krb5kdc[21870](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935663, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.102.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 06:07:43.718102 21883 init.cc:377] Logged in from keytab as kudu/127.11.102.62@KRBTEST.COM (short username kudu)
I20260620 06:07:43.718997 21883 webserver.cc:460] Webserver started at http://127.11.102.62:45497/ using document root <none> and password file <none>
I20260620 06:07:43.719548 21883 fs_manager.cc:359] Metadata directory not provided
I20260620 06:07:43.719663 21883 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:07:43.719892 21883 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 06:07:43.721846 21883 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781935603120653-11672-0/minicluster-data/master-0/data/instance:
uuid: "b71d23ee5a044402b769ed9b36de8a06"
format_stamp: "Formatted at 2026-06-20 06:07:43 on dist-test-slave-9hm5"
server_key: "9a7a253b4a765fd53827448058e26301"
server_key_iv: "f394e4962f449d22f56b97f4fa5195b5"
server_key_version: "encryptionkey@0"
I20260620 06:07:43.722378 21883 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781935603120653-11672-0/minicluster-data/master-0/wal/instance:
uuid: "b71d23ee5a044402b769ed9b36de8a06"
format_stamp: "Formatted at 2026-06-20 06:07:43 on dist-test-slave-9hm5"
server_key: "9a7a253b4a765fd53827448058e26301"
server_key_iv: "f394e4962f449d22f56b97f4fa5195b5"
server_key_version: "encryptionkey@0"
I20260620 06:07:43.725912 21883 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260620 06:07:43.728227 21898 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:07:43.729455 21883 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 06:07:43.729621 21883 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781935603120653-11672-0/minicluster-data/master-0/data,/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781935603120653-11672-0/minicluster-data/master-0/wal
uuid: "b71d23ee5a044402b769ed9b36de8a06"
format_stamp: "Formatted at 2026-06-20 06:07:43 on dist-test-slave-9hm5"
server_key: "9a7a253b4a765fd53827448058e26301"
server_key_iv: "f394e4962f449d22f56b97f4fa5195b5"
server_key_version: "encryptionkey@0"
I20260620 06:07:43.729736 21883 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781935603120653-11672-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781935603120653-11672-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781935603120653-11672-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:07:43.744094 21883 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:07:43.747021 21883 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:07:43.747256 21883 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:07:43.753847 21883 rpc_server.cc:307] RPC server started. Bound to: 127.11.102.62:46701
I20260620 06:07:43.754213 21950 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.102.62:46701 every 8 connection(s)
I20260620 06:07:43.754828 21883 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781935603120653-11672-0/minicluster-data/master-0/data/info.pb
I20260620 06:07:43.757501 21951 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 06:07:43.759120 11672 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9HMp43/build/debug/bin/kudu as pid 21883
I20260620 06:07:43.759255 11672 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781935603120653-11672-0/minicluster-data/master-0/wal/instance
I20260620 06:07:43.759531 11672 external_mini_cluster.cc:1428] Setting key b0500f11605c75ff120d6eaa72c8492b
I20260620 06:07:43.764073 21951 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b71d23ee5a044402b769ed9b36de8a06: Bootstrap starting.
I20260620 06:07:43.766587 21951 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b71d23ee5a044402b769ed9b36de8a06: Neither blocks nor log segments found. Creating new log.
I20260620 06:07:43.767465 21951 log.cc:824] T 00000000000000000000000000000000 P b71d23ee5a044402b769ed9b36de8a06: Log is configured to *not* fsync() on all Append() calls
Jun 20 06:07:43 dist-test-slave-9hm5 krb5kdc[21870](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935663, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.102.62@KRBTEST.COM
I20260620 06:07:43.769392 21951 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b71d23ee5a044402b769ed9b36de8a06: No bootstrap required, opened a new log
I20260620 06:07:43.774133 21951 raft_consensus.cc:348] T 00000000000000000000000000000000 P b71d23ee5a044402b769ed9b36de8a06 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b71d23ee5a044402b769ed9b36de8a06" member_type: VOTER last_known_addr { host: "127.11.102.62" port: 46701 } }
I20260620 06:07:43.774395 21951 raft_consensus.cc:374] T 00000000000000000000000000000000 P b71d23ee5a044402b769ed9b36de8a06 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 06:07:43.774438 21954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:07:43.760941 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56476 (local address 127.11.102.62:46701)
0620 06:07:43.761359 (+ 418us) server_negotiation.cc:206] Beginning negotiation
0620 06:07:43.761378 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0620 06:07:43.761419 (+ 41us) server_negotiation.cc:407] Connection header received
0620 06:07:43.762237 (+ 818us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:07:43.762261 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:07:43.762679 (+ 418us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:07:43.763039 (+ 360us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:07:43.763703 (+ 664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:07:43.765106 (+ 1403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:07:43.765929 (+ 823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:07:43.766618 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:07:43.766785 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:07:43.769278 (+ 2493us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:07:43.769284 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:07:43.769300 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:07:43.769329 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:07:43.771250 (+ 1921us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:07:43.771752 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:07:43.771759 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:07:43.771767 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:07:43.771938 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:07:43.772265 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:07:43.772268 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:07:43.772270 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:07:43.772766 (+ 496us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:07:43.772968 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0620 06:07:43.773179 (+ 211us) server_negotiation.cc:299] Negotiation successful
0620 06:07:43.773482 (+ 303us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":282,"thread_start_us":126,"threads_started":1}
I20260620 06:07:43.774458 21951 raft_consensus.cc:729] T 00000000000000000000000000000000 P b71d23ee5a044402b769ed9b36de8a06 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b71d23ee5a044402b769ed9b36de8a06, State: Initialized, Role: FOLLOWER
I20260620 06:07:43.774901 21951 consensus_queue.cc:260] T 00000000000000000000000000000000 P b71d23ee5a044402b769ed9b36de8a06 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b71d23ee5a044402b769ed9b36de8a06" member_type: VOTER last_known_addr { host: "127.11.102.62" port: 46701 } }
I20260620 06:07:43.775075 21951 raft_consensus.cc:388] T 00000000000000000000000000000000 P b71d23ee5a044402b769ed9b36de8a06 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 06:07:43.775166 21951 raft_consensus.cc:482] T 00000000000000000000000000000000 P b71d23ee5a044402b769ed9b36de8a06 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 06:07:43.775264 21951 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b71d23ee5a044402b769ed9b36de8a06 [term 0 FOLLOWER]: Advancing to term 1
I20260620 06:07:43.776435 21951 raft_consensus.cc:504] T 00000000000000000000000000000000 P b71d23ee5a044402b769ed9b36de8a06 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b71d23ee5a044402b769ed9b36de8a06" member_type: VOTER last_known_addr { host: "127.11.102.62" port: 46701 } }
I20260620 06:07:43.776823 21951 leader_election.cc:302] T 00000000000000000000000000000000 P b71d23ee5a044402b769ed9b36de8a06 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b71d23ee5a044402b769ed9b36de8a06; no voters:
I20260620 06:07:43.777168 21951 leader_election.cc:288] T 00000000000000000000000000000000 P b71d23ee5a044402b769ed9b36de8a06 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 06:07:43.777298 21956 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b71d23ee5a044402b769ed9b36de8a06 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 06:07:43.777527 21956 raft_consensus.cc:686] T 00000000000000000000000000000000 P b71d23ee5a044402b769ed9b36de8a06 [term 1 LEADER]: Becoming Leader. State: Replica: b71d23ee5a044402b769ed9b36de8a06, State: Running, Role: LEADER
I20260620 06:07:43.777853 21956 consensus_queue.cc:237] T 00000000000000000000000000000000 P b71d23ee5a044402b769ed9b36de8a06 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b71d23ee5a044402b769ed9b36de8a06" member_type: VOTER last_known_addr { host: "127.11.102.62" port: 46701 } }
I20260620 06:07:43.778424 21951 sys_catalog.cc:564] T 00000000000000000000000000000000 P b71d23ee5a044402b769ed9b36de8a06 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 06:07:43.779443 21957 sys_catalog.cc:455] T 00000000000000000000000000000000 P b71d23ee5a044402b769ed9b36de8a06 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b71d23ee5a044402b769ed9b36de8a06" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b71d23ee5a044402b769ed9b36de8a06" member_type: VOTER last_known_addr { host: "127.11.102.62" port: 46701 } } }
I20260620 06:07:43.779572 21957 sys_catalog.cc:458] T 00000000000000000000000000000000 P b71d23ee5a044402b769ed9b36de8a06 [sys.catalog]: This master's current role is: LEADER
I20260620 06:07:43.779982 21958 sys_catalog.cc:455] T 00000000000000000000000000000000 P b71d23ee5a044402b769ed9b36de8a06 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b71d23ee5a044402b769ed9b36de8a06. Latest consensus state: current_term: 1 leader_uuid: "b71d23ee5a044402b769ed9b36de8a06" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b71d23ee5a044402b769ed9b36de8a06" member_type: VOTER last_known_addr { host: "127.11.102.62" port: 46701 } } }
I20260620 06:07:43.780154 21958 sys_catalog.cc:458] T 00000000000000000000000000000000 P b71d23ee5a044402b769ed9b36de8a06 [sys.catalog]: This master's current role is: LEADER
I20260620 06:07:43.780632 21965 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 06:07:43.783286 21965 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 06:07:43.788702 21965 catalog_manager.cc:1347] Generated new cluster ID: b7f1abcdb0c940dab2e38143401a5712
I20260620 06:07:43.788769 21965 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 06:07:43.799136 21965 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 06:07:43.800325 21965 catalog_manager.cc:1504] Loading token signing keys...
I20260620 06:07:43.806650 21965 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b71d23ee5a044402b769ed9b36de8a06: Generated new TSK 0
I20260620 06:07:43.807328 21965 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.11.102.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.102.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.102.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781935603120653-11672-0/krb5kdc/kudu_127.11.102.1.keytab.
Entry for principal kudu/127.11.102.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781935603120653-11672-0/krb5kdc/kudu_127.11.102.1.keytab.
I20260620 06:07:43.834046 11672 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9HMp43/build/debug/bin/kudu
/tmp/dist-test-task9HMp43/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781935603120653-11672-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781935603120653-11672-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781935603120653-11672-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781935603120653-11672-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.102.1:0
--local_ip_for_outbound_sockets=127.11.102.1
--webserver_interface=127.11.102.1
--webserver_port=0
--tserver_master_addrs=127.11.102.62:46701
--keytab_file=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781935603120653-11672-0/krb5kdc/kudu_127.11.102.1.keytab
--principal=kudu/127.11.102.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.102.20:37729
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781935603120653-11672-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781935603120653-11672-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781935603120653-11672-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 06:07:43.941627 21977 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:07:43.941893 21977 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:07:43.941969 21977 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:07:43.945883 21977 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:07:43.945962 21977 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:07:43.946058 21977 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.102.1
I20260620 06:07:43.951089 21977 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.102.20:37729
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781935603120653-11672-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781935603120653-11672-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781935603120653-11672-0/krb5kdc/kudu_127.11.102.1.keytab
--principal=kudu/127.11.102.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.102.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781935603120653-11672-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.11.102.1
--webserver_port=0
--tserver_master_addrs=127.11.102.62:46701
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21977
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.102.1
--log_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781935603120653-11672-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 05:28:48 UTC on ccycloud.quasar-fixhms.root.comops.site
build id 4457020
I20260620 06:07:43.952088 21977 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:07:43.953058 21977 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 06:07:43.962329 21985 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:07:43.962397 21982 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:07:43.962533 21983 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:07:43.962549 21977 server_base.cc:1034] running on GCE node
I20260620 06:07:43.963084 21977 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:07:43.963737 21977 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:07:43.964919 21977 hybrid_clock.cc:648] HybridClock initialized: now 1781935663964889 us; error 49 us; skew 500 ppm
Jun 20 06:07:43 dist-test-slave-9hm5 krb5kdc[21870](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935663, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.102.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 06:07:43.967597 21977 init.cc:377] Logged in from keytab as kudu/127.11.102.1@KRBTEST.COM (short username kudu)
I20260620 06:07:43.968453 21977 webserver.cc:460] Webserver started at http://127.11.102.1:36475/ using document root <none> and password file <none>
I20260620 06:07:43.968966 21977 fs_manager.cc:359] Metadata directory not provided
I20260620 06:07:43.969026 21977 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:07:43.969185 21977 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 06:07:43.971100 21977 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781935603120653-11672-0/minicluster-data/ts-0/data/instance:
uuid: "2cf36ed63d0f4a6e8c002166660f0867"
format_stamp: "Formatted at 2026-06-20 06:07:43 on dist-test-slave-9hm5"
server_key: "fdf6fa4acb39291b324c151863cc41e1"
server_key_iv: "ed9c193faae8693f42e861cf3e796d1d"
server_key_version: "encryptionkey@0"
I20260620 06:07:43.971706 21977 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781935603120653-11672-0/minicluster-data/ts-0/wal/instance:
uuid: "2cf36ed63d0f4a6e8c002166660f0867"
format_stamp: "Formatted at 2026-06-20 06:07:43 on dist-test-slave-9hm5"
server_key: "fdf6fa4acb39291b324c151863cc41e1"
server_key_iv: "ed9c193faae8693f42e861cf3e796d1d"
server_key_version: "encryptionkey@0"
I20260620 06:07:43.975253 21977 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.002s
I20260620 06:07:43.977690 21992 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:07:43.978878 21977 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260620 06:07:43.979074 21977 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781935603120653-11672-0/minicluster-data/ts-0/data,/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781935603120653-11672-0/minicluster-data/ts-0/wal
uuid: "2cf36ed63d0f4a6e8c002166660f0867"
format_stamp: "Formatted at 2026-06-20 06:07:43 on dist-test-slave-9hm5"
server_key: "fdf6fa4acb39291b324c151863cc41e1"
server_key_iv: "ed9c193faae8693f42e861cf3e796d1d"
server_key_version: "encryptionkey@0"
I20260620 06:07:43.979212 21977 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781935603120653-11672-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781935603120653-11672-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781935603120653-11672-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:07:43.996695 21977 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:07:43.999944 21977 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:07:44.000157 21977 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:07:44.000738 21977 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 06:07:44.001540 21977 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 06:07:44.001608 21977 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 06:07:44.001677 21977 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 06:07:44.001724 21977 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 06:07:44.010325 21977 rpc_server.cc:307] RPC server started. Bound to: 127.11.102.1:39933
I20260620 06:07:44.010354 22105 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.102.1:39933 every 8 connection(s)
I20260620 06:07:44.011246 21977 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781935603120653-11672-0/minicluster-data/ts-0/data/info.pb
I20260620 06:07:44.019898 11672 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9HMp43/build/debug/bin/kudu as pid 21977
I20260620 06:07:44.020035 11672 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781935603120653-11672-0/minicluster-data/ts-0/wal/instance
I20260620 06:07:44.020358 11672 external_mini_cluster.cc:1428] Setting key d7dcd060e113033118663f3249e66bcb
Jun 20 06:07:44 dist-test-slave-9hm5 krb5kdc[21870](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935663, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.102.1@KRBTEST.COM for kudu/127.11.102.62@KRBTEST.COM
I20260620 06:07:44.026665 21954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:07:44.013183 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.102.1:34715 (local address 127.11.102.62:46701)
0620 06:07:44.013345 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0620 06:07:44.013350 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 06:07:44.014198 (+ 848us) server_negotiation.cc:407] Connection header received
0620 06:07:44.015491 (+ 1293us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:07:44.015494 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:07:44.015538 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:07:44.015680 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:07:44.017089 (+ 1409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:07:44.017884 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:07:44.019667 (+ 1783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:07:44.020020 (+ 353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:07:44.020131 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:07:44.022817 (+ 2686us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:07:44.022823 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:07:44.022830 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:07:44.022864 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:07:44.024469 (+ 1605us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:07:44.024985 (+ 516us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:07:44.024991 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:07:44.024996 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:07:44.025086 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:07:44.025491 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:07:44.025494 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:07:44.025495 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:07:44.025671 (+ 176us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:07:44.025749 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0620 06:07:44.026363 (+ 614us) server_negotiation.cc:299] Negotiation successful
0620 06:07:44.026523 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260620 06:07:44.027769 22108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:07:44.013584 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.102.62:46701 (local address 127.11.102.1:34715)
0620 06:07:44.014050 (+ 466us) negotiation.cc:107] Waiting for socket to connect
0620 06:07:44.014092 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0620 06:07:44.015306 (+ 1214us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:07:44.015840 (+ 534us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:07:44.015846 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:07:44.016218 (+ 372us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:07:44.016902 (+ 684us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:07:44.016944 (+ 42us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:07:44.018031 (+ 1087us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:07:44.018035 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:07:44.019505 (+ 1470us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:07:44.019517 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:07:44.020178 (+ 661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:07:44.020181 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:07:44.020322 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:07:44.021012 (+ 690us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:07:44.021052 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:07:44.022660 (+ 1608us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:07:44.024591 (+ 1931us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:07:44.024597 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:07:44.024612 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:07:44.024867 (+ 255us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:07:44.025192 (+ 325us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:07:44.025197 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:07:44.025202 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:07:44.025376 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:07:44.025774 (+ 398us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:07:44.025784 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:07:44.026104 (+ 320us) client_negotiation.cc:769] Sending connection context
0620 06:07:44.026332 (+ 228us) client_negotiation.cc:240] Negotiation successful
0620 06:07:44.026731 (+ 399us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":272,"thread_start_us":114,"threads_started":1}
I20260620 06:07:44.029134 22106 heartbeater.cc:347] Connected to a master server at 127.11.102.62:46701
I20260620 06:07:44.029384 22106 heartbeater.cc:464] Registering TS with master...
I20260620 06:07:44.029881 22106 heartbeater.cc:511] Master 127.11.102.62:46701 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.11.102.2@KRBTEST.COM; defaulting to no policy
I20260620 06:07:44.031391 21915 ts_manager.cc:194] Registered new tserver with Master: 2cf36ed63d0f4a6e8c002166660f0867 (127.11.102.1:39933)
I20260620 06:07:44.032730 21915 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.102.1@KRBTEST.COM'} at 127.11.102.1:34715
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.102.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.102.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781935603120653-11672-0/krb5kdc/kudu_127.11.102.2.keytab.
Entry for principal kudu/127.11.102.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781935603120653-11672-0/krb5kdc/kudu_127.11.102.2.keytab.
I20260620 06:07:44.046392 11672 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9HMp43/build/debug/bin/kudu
/tmp/dist-test-task9HMp43/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781935603120653-11672-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781935603120653-11672-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781935603120653-11672-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781935603120653-11672-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.102.2:0
--local_ip_for_outbound_sockets=127.11.102.2
--webserver_interface=127.11.102.2
--webserver_port=0
--tserver_master_addrs=127.11.102.62:46701
--keytab_file=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781935603120653-11672-0/krb5kdc/kudu_127.11.102.2.keytab
--principal=kudu/127.11.102.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.102.20:37729
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781935603120653-11672-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781935603120653-11672-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781935603120653-11672-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 06:07:44.153831 22111 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:07:44.154115 22111 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:07:44.154189 22111 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:07:44.158177 22111 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:07:44.158262 22111 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:07:44.158352 22111 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.102.2
I20260620 06:07:44.163480 22111 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.102.20:37729
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781935603120653-11672-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781935603120653-11672-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781935603120653-11672-0/krb5kdc/kudu_127.11.102.2.keytab
--principal=kudu/127.11.102.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.102.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781935603120653-11672-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.11.102.2
--webserver_port=0
--tserver_master_addrs=127.11.102.62:46701
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22111
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.102.2
--log_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781935603120653-11672-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 05:28:48 UTC on ccycloud.quasar-fixhms.root.comops.site
build id 4457020
I20260620 06:07:44.164536 22111 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:07:44.165534 22111 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 06:07:44.174624 22116 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:07:44.174639 22119 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:07:44.174671 22117 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:07:44.174850 22111 server_base.cc:1034] running on GCE node
I20260620 06:07:44.175304 22111 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:07:44.175971 22111 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:07:44.177152 22111 hybrid_clock.cc:648] HybridClock initialized: now 1781935664177114 us; error 50 us; skew 500 ppm
Jun 20 06:07:44 dist-test-slave-9hm5 krb5kdc[21870](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935664, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.102.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 06:07:44.179978 22111 init.cc:377] Logged in from keytab as kudu/127.11.102.2@KRBTEST.COM (short username kudu)
I20260620 06:07:44.180828 22111 webserver.cc:460] Webserver started at http://127.11.102.2:46757/ using document root <none> and password file <none>
I20260620 06:07:44.181324 22111 fs_manager.cc:359] Metadata directory not provided
I20260620 06:07:44.181380 22111 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:07:44.181538 22111 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 06:07:44.183571 22111 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781935603120653-11672-0/minicluster-data/ts-1/data/instance:
uuid: "20efde36d9dc44a6b8c09f28eb3fed1e"
format_stamp: "Formatted at 2026-06-20 06:07:44 on dist-test-slave-9hm5"
server_key: "a78881d2a4c5d2d0ce35e71571ac9338"
server_key_iv: "340d2ea42cc0e3a5dd34f9583a9e977b"
server_key_version: "encryptionkey@0"
I20260620 06:07:44.184127 22111 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781935603120653-11672-0/minicluster-data/ts-1/wal/instance:
uuid: "20efde36d9dc44a6b8c09f28eb3fed1e"
format_stamp: "Formatted at 2026-06-20 06:07:44 on dist-test-slave-9hm5"
server_key: "a78881d2a4c5d2d0ce35e71571ac9338"
server_key_iv: "340d2ea42cc0e3a5dd34f9583a9e977b"
server_key_version: "encryptionkey@0"
I20260620 06:07:44.188035 22111 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.002s
I20260620 06:07:44.190697 22126 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:07:44.192093 22111 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 06:07:44.192209 22111 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781935603120653-11672-0/minicluster-data/ts-1/data,/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781935603120653-11672-0/minicluster-data/ts-1/wal
uuid: "20efde36d9dc44a6b8c09f28eb3fed1e"
format_stamp: "Formatted at 2026-06-20 06:07:44 on dist-test-slave-9hm5"
server_key: "a78881d2a4c5d2d0ce35e71571ac9338"
server_key_iv: "340d2ea42cc0e3a5dd34f9583a9e977b"
server_key_version: "encryptionkey@0"
I20260620 06:07:44.192325 22111 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781935603120653-11672-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781935603120653-11672-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781935603120653-11672-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:07:44.216148 22111 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:07:44.219548 22111 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:07:44.219785 22111 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:07:44.220389 22111 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 06:07:44.221215 22111 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 06:07:44.221283 22111 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 06:07:44.221355 22111 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 06:07:44.221411 22111 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 06:07:44.230182 22111 rpc_server.cc:307] RPC server started. Bound to: 127.11.102.2:39443
I20260620 06:07:44.230206 22239 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.102.2:39443 every 8 connection(s)
I20260620 06:07:44.231061 22111 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781935603120653-11672-0/minicluster-data/ts-1/data/info.pb
I20260620 06:07:44.232187 11672 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9HMp43/build/debug/bin/kudu as pid 22111
I20260620 06:07:44.232304 11672 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781935603120653-11672-0/minicluster-data/ts-1/wal/instance
I20260620 06:07:44.232623 11672 external_mini_cluster.cc:1428] Setting key 8da2abf88eeff8fae41fcd3f5b86b912
Jun 20 06:07:44 dist-test-slave-9hm5 krb5kdc[21870](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935664, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.102.2@KRBTEST.COM for kudu/127.11.102.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.11.102.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.102.3@KRBTEST.COM" created.
I20260620 06:07:44.246621 21954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:07:44.233042 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.102.2:36449 (local address 127.11.102.62:46701)
0620 06:07:44.233161 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0620 06:07:44.233164 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 06:07:44.234233 (+ 1069us) server_negotiation.cc:407] Connection header received
0620 06:07:44.236304 (+ 2071us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:07:44.236308 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:07:44.236360 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:07:44.236442 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:07:44.237598 (+ 1156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:07:44.238486 (+ 888us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:07:44.239665 (+ 1179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:07:44.240095 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:07:44.240232 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:07:44.242696 (+ 2464us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:07:44.242703 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:07:44.242708 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:07:44.242745 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:07:44.244365 (+ 1620us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:07:44.244942 (+ 577us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:07:44.244945 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:07:44.244946 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:07:44.245034 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:07:44.245389 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:07:44.245392 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:07:44.245393 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:07:44.245621 (+ 228us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:07:44.245728 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0620 06:07:44.246293 (+ 565us) server_negotiation.cc:299] Negotiation successful
0620 06:07:44.246420 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260620 06:07:44.247506 22242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:07:44.233518 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.102.62:46701 (local address 127.11.102.2:36449)
0620 06:07:44.234053 (+ 535us) negotiation.cc:107] Waiting for socket to connect
0620 06:07:44.234128 (+ 75us) client_negotiation.cc:174] Beginning negotiation
0620 06:07:44.236089 (+ 1961us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:07:44.236628 (+ 539us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:07:44.236635 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:07:44.236990 (+ 355us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:07:44.237455 (+ 465us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:07:44.237468 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:07:44.238632 (+ 1164us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:07:44.238635 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:07:44.239506 (+ 871us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:07:44.239515 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:07:44.240288 (+ 773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:07:44.240292 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:07:44.240491 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:07:44.241110 (+ 619us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:07:44.241126 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:07:44.242522 (+ 1396us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:07:44.244502 (+ 1980us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:07:44.244506 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:07:44.244518 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:07:44.244787 (+ 269us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:07:44.245138 (+ 351us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:07:44.245141 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:07:44.245142 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:07:44.245296 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:07:44.245753 (+ 457us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:07:44.245761 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:07:44.246020 (+ 259us) client_negotiation.cc:769] Sending connection context
0620 06:07:44.246230 (+ 210us) client_negotiation.cc:240] Negotiation successful
0620 06:07:44.246566 (+ 336us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":293,"thread_start_us":145,"threads_started":1}
I20260620 06:07:44.248746 22240 heartbeater.cc:347] Connected to a master server at 127.11.102.62:46701
I20260620 06:07:44.248987 22240 heartbeater.cc:464] Registering TS with master...
I20260620 06:07:44.249593 22240 heartbeater.cc:511] Master 127.11.102.62:46701 requested a full tablet report, sending...
I20260620 06:07:44.250775 21915 ts_manager.cc:194] Registered new tserver with Master: 20efde36d9dc44a6b8c09f28eb3fed1e (127.11.102.2:39443)
I20260620 06:07:44.251663 21915 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.102.2@KRBTEST.COM'} at 127.11.102.2:36449
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.102.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781935603120653-11672-0/krb5kdc/kudu_127.11.102.3.keytab.
Entry for principal kudu/127.11.102.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781935603120653-11672-0/krb5kdc/kudu_127.11.102.3.keytab.
I20260620 06:07:44.260429 11672 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9HMp43/build/debug/bin/kudu
/tmp/dist-test-task9HMp43/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781935603120653-11672-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781935603120653-11672-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781935603120653-11672-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781935603120653-11672-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.102.3:0
--local_ip_for_outbound_sockets=127.11.102.3
--webserver_interface=127.11.102.3
--webserver_port=0
--tserver_master_addrs=127.11.102.62:46701
--keytab_file=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781935603120653-11672-0/krb5kdc/kudu_127.11.102.3.keytab
--principal=kudu/127.11.102.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.102.20:37729
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781935603120653-11672-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781935603120653-11672-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781935603120653-11672-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 06:07:44.368432 22245 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:07:44.368669 22245 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:07:44.368710 22245 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:07:44.372648 22245 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:07:44.372704 22245 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:07:44.372759 22245 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.102.3
I20260620 06:07:44.377995 22245 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.102.20:37729
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781935603120653-11672-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781935603120653-11672-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781935603120653-11672-0/krb5kdc/kudu_127.11.102.3.keytab
--principal=kudu/127.11.102.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.102.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781935603120653-11672-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.11.102.3
--webserver_port=0
--tserver_master_addrs=127.11.102.62:46701
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22245
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.102.3
--log_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781935603120653-11672-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 05:28:48 UTC on ccycloud.quasar-fixhms.root.comops.site
build id 4457020
I20260620 06:07:44.379009 22245 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:07:44.380084 22245 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 06:07:44.389720 22253 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:07:44.389781 22251 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:07:44.389750 22250 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:07:44.389904 22245 server_base.cc:1034] running on GCE node
I20260620 06:07:44.390462 22245 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:07:44.391072 22245 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:07:44.392246 22245 hybrid_clock.cc:648] HybridClock initialized: now 1781935664392216 us; error 42 us; skew 500 ppm
Jun 20 06:07:44 dist-test-slave-9hm5 krb5kdc[21870](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935664, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.102.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 06:07:44.394949 22245 init.cc:377] Logged in from keytab as kudu/127.11.102.3@KRBTEST.COM (short username kudu)
I20260620 06:07:44.395792 22245 webserver.cc:460] Webserver started at http://127.11.102.3:36529/ using document root <none> and password file <none>
I20260620 06:07:44.396378 22245 fs_manager.cc:359] Metadata directory not provided
I20260620 06:07:44.396440 22245 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:07:44.396598 22245 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 06:07:44.398558 22245 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781935603120653-11672-0/minicluster-data/ts-2/data/instance:
uuid: "a347ab2184e54b1ca341ca15bc069b9d"
format_stamp: "Formatted at 2026-06-20 06:07:44 on dist-test-slave-9hm5"
server_key: "23a9a320d81afede3ce10e80d10cdf6c"
server_key_iv: "65844f35d00e40d8fefa6fcf63e9497f"
server_key_version: "encryptionkey@0"
I20260620 06:07:44.399055 22245 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781935603120653-11672-0/minicluster-data/ts-2/wal/instance:
uuid: "a347ab2184e54b1ca341ca15bc069b9d"
format_stamp: "Formatted at 2026-06-20 06:07:44 on dist-test-slave-9hm5"
server_key: "23a9a320d81afede3ce10e80d10cdf6c"
server_key_iv: "65844f35d00e40d8fefa6fcf63e9497f"
server_key_version: "encryptionkey@0"
I20260620 06:07:44.402561 22245 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260620 06:07:44.404970 22260 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:07:44.406198 22245 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260620 06:07:44.406340 22245 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781935603120653-11672-0/minicluster-data/ts-2/data,/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781935603120653-11672-0/minicluster-data/ts-2/wal
uuid: "a347ab2184e54b1ca341ca15bc069b9d"
format_stamp: "Formatted at 2026-06-20 06:07:44 on dist-test-slave-9hm5"
server_key: "23a9a320d81afede3ce10e80d10cdf6c"
server_key_iv: "65844f35d00e40d8fefa6fcf63e9497f"
server_key_version: "encryptionkey@0"
I20260620 06:07:44.406451 22245 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781935603120653-11672-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781935603120653-11672-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781935603120653-11672-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:07:44.417543 22245 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:07:44.420550 22245 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:07:44.420747 22245 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:07:44.421259 22245 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 06:07:44.421995 22245 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 06:07:44.422061 22245 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 06:07:44.422127 22245 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 06:07:44.422179 22245 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 06:07:44.430815 22245 rpc_server.cc:307] RPC server started. Bound to: 127.11.102.3:46621
I20260620 06:07:44.430832 22373 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.102.3:46621 every 8 connection(s)
I20260620 06:07:44.431721 22245 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781935603120653-11672-0/minicluster-data/ts-2/data/info.pb
I20260620 06:07:44.436262 11672 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9HMp43/build/debug/bin/kudu as pid 22245
I20260620 06:07:44.436393 11672 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781935603120653-11672-0/minicluster-data/ts-2/wal/instance
I20260620 06:07:44.436713 11672 external_mini_cluster.cc:1428] Setting key 0983890af230d4f416cb24aafb26f546
Jun 20 06:07:44 dist-test-slave-9hm5 krb5kdc[21870](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935664, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.102.3@KRBTEST.COM for kudu/127.11.102.62@KRBTEST.COM
I20260620 06:07:44.445849 21954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:07:44.433604 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.102.3:59631 (local address 127.11.102.62:46701)
0620 06:07:44.433767 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0620 06:07:44.433770 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 06:07:44.434658 (+ 888us) server_negotiation.cc:407] Connection header received
0620 06:07:44.436228 (+ 1570us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:07:44.436231 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:07:44.436295 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:07:44.436386 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:07:44.438018 (+ 1632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:07:44.438762 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:07:44.439913 (+ 1151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:07:44.440117 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:07:44.440194 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:07:44.442670 (+ 2476us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:07:44.442673 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:07:44.442675 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:07:44.442697 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:07:44.444045 (+ 1348us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:07:44.444493 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:07:44.444495 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:07:44.444497 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:07:44.444567 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:07:44.444903 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:07:44.444906 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:07:44.444907 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:07:44.445080 (+ 173us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:07:44.445165 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0620 06:07:44.445630 (+ 465us) server_negotiation.cc:299] Negotiation successful
0620 06:07:44.445725 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260620 06:07:44.446625 22376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:07:44.434042 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.102.62:46701 (local address 127.11.102.3:59631)
0620 06:07:44.434501 (+ 459us) negotiation.cc:107] Waiting for socket to connect
0620 06:07:44.434542 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0620 06:07:44.435860 (+ 1318us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:07:44.436693 (+ 833us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:07:44.436700 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:07:44.437170 (+ 470us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:07:44.437836 (+ 666us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:07:44.437856 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:07:44.438885 (+ 1029us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:07:44.438887 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:07:44.439816 (+ 929us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:07:44.439825 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:07:44.440254 (+ 429us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:07:44.440257 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:07:44.440391 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:07:44.441047 (+ 656us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:07:44.441063 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:07:44.442497 (+ 1434us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:07:44.444144 (+ 1647us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:07:44.444148 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:07:44.444159 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:07:44.444409 (+ 250us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:07:44.444652 (+ 243us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:07:44.444654 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:07:44.444656 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:07:44.444802 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:07:44.445203 (+ 401us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:07:44.445210 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:07:44.445425 (+ 215us) client_negotiation.cc:769] Sending connection context
0620 06:07:44.445587 (+ 162us) client_negotiation.cc:240] Negotiation successful
0620 06:07:44.445832 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":269,"thread_start_us":116,"threads_started":1}
I20260620 06:07:44.447703 22374 heartbeater.cc:347] Connected to a master server at 127.11.102.62:46701
I20260620 06:07:44.447934 22374 heartbeater.cc:464] Registering TS with master...
I20260620 06:07:44.448400 22374 heartbeater.cc:511] Master 127.11.102.62:46701 requested a full tablet report, sending...
I20260620 06:07:44.449474 21915 ts_manager.cc:194] Registered new tserver with Master: a347ab2184e54b1ca341ca15bc069b9d (127.11.102.3:46621)
I20260620 06:07:44.450069 21915 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.102.3@KRBTEST.COM'} at 127.11.102.3:59631
I20260620 06:07:44.450434 11672 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 06:07:44.458906 21954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:07:44.452244 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56492 (local address 127.11.102.62:46701)
0620 06:07:44.452390 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0620 06:07:44.452395 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 06:07:44.452492 (+ 97us) server_negotiation.cc:407] Connection header received
0620 06:07:44.452578 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:07:44.452580 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:07:44.452614 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:07:44.452672 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:07:44.453415 (+ 743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:07:44.453913 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:07:44.454707 (+ 794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:07:44.454916 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:07:44.454994 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:07:44.456360 (+ 1366us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:07:44.456363 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:07:44.456365 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:07:44.456383 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:07:44.457524 (+ 1141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:07:44.457886 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:07:44.457889 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:07:44.457890 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:07:44.457954 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:07:44.458172 (+ 218us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:07:44.458174 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:07:44.458175 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:07:44.458350 (+ 175us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:07:44.458447 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0620 06:07:44.458624 (+ 177us) server_negotiation.cc:299] Negotiation successful
0620 06:07:44.458703 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260620 06:07:44.461692 21915 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56492:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 06:07:44.464625 21915 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 06:07:44.478900 22386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:07:44.472580 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.102.1:39933 (local address 127.0.0.1:52276)
0620 06:07:44.473529 (+ 949us) negotiation.cc:107] Waiting for socket to connect
0620 06:07:44.473582 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0620 06:07:44.473993 (+ 411us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:07:44.474424 (+ 431us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:07:44.474430 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:07:44.474466 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 06:07:44.474709 (+ 243us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:07:44.474718 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:07:44.475901 (+ 1183us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:07:44.475905 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:07:44.477325 (+ 1420us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:07:44.477335 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:07:44.478353 (+ 1018us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:07:44.478357 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:07:44.478460 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:07:44.478503 (+ 43us) client_negotiation.cc:769] Sending connection context
0620 06:07:44.478604 (+ 101us) client_negotiation.cc:240] Negotiation successful
0620 06:07:44.478712 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":779,"mutex_wait_us":135,"spinlock_wait_cycles":5760,"thread_start_us":82,"threads_started":1}
I20260620 06:07:44.478992 22387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:07:44.472673 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52276 (local address 127.11.102.1:39933)
0620 06:07:44.473280 (+ 607us) server_negotiation.cc:206] Beginning negotiation
0620 06:07:44.473293 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 06:07:44.473668 (+ 375us) server_negotiation.cc:407] Connection header received
0620 06:07:44.474125 (+ 457us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:07:44.474131 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:07:44.474277 (+ 146us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:07:44.474435 (+ 158us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 06:07:44.474822 (+ 387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:07:44.475771 (+ 949us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:07:44.477428 (+ 1657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:07:44.478226 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:07:44.478341 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:07:44.478482 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0620 06:07:44.478683 (+ 201us) server_negotiation.cc:299] Negotiation successful
0620 06:07:44.478800 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":481,"thread_start_us":135,"threads_started":1}
I20260620 06:07:44.479461 22385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:07:44.473322 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.102.2:39443 (local address 127.0.0.1:43012)
0620 06:07:44.473824 (+ 502us) negotiation.cc:107] Waiting for socket to connect
0620 06:07:44.473837 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 06:07:44.473922 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:07:44.475103 (+ 1181us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:07:44.475106 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:07:44.475123 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 06:07:44.475211 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:07:44.475221 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:07:44.476630 (+ 1409us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:07:44.476633 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:07:44.477651 (+ 1018us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:07:44.477657 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:07:44.478569 (+ 912us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:07:44.478572 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:07:44.478684 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:07:44.478709 (+ 25us) client_negotiation.cc:769] Sending connection context
0620 06:07:44.478833 (+ 124us) client_negotiation.cc:240] Negotiation successful
0620 06:07:44.478904 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":418,"mutex_wait_us":3,"thread_start_us":83,"threads_started":1}
I20260620 06:07:44.479822 22390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:07:44.473473 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43012 (local address 127.11.102.2:39443)
0620 06:07:44.474666 (+ 1193us) server_negotiation.cc:206] Beginning negotiation
0620 06:07:44.474674 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 06:07:44.474691 (+ 17us) server_negotiation.cc:407] Connection header received
0620 06:07:44.474781 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:07:44.474785 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:07:44.474952 (+ 167us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:07:44.475099 (+ 147us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 06:07:44.475334 (+ 235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:07:44.476456 (+ 1122us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:07:44.477759 (+ 1303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:07:44.478474 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:07:44.478594 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:07:44.478727 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0620 06:07:44.478979 (+ 252us) server_negotiation.cc:299] Negotiation successful
0620 06:07:44.479651 (+ 672us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1094,"thread_start_us":99,"threads_started":1}
I20260620 06:07:44.480255 22389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:07:44.472661 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.102.3:46621 (local address 127.0.0.1:41520)
0620 06:07:44.473653 (+ 992us) negotiation.cc:107] Waiting for socket to connect
0620 06:07:44.473668 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 06:07:44.473783 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:07:44.474425 (+ 642us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:07:44.474430 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:07:44.474467 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 06:07:44.474641 (+ 174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:07:44.474652 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:07:44.476259 (+ 1607us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:07:44.476262 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:07:44.477251 (+ 989us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:07:44.477262 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:07:44.478548 (+ 1286us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:07:44.478553 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:07:44.478859 (+ 306us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:07:44.478879 (+ 20us) client_negotiation.cc:769] Sending connection context
0620 06:07:44.478951 (+ 72us) client_negotiation.cc:240] Negotiation successful
0620 06:07:44.479025 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":925,"mutex_wait_us":211,"spinlock_wait_cycles":7296,"thread_start_us":103,"threads_started":1}
I20260620 06:07:44.480755 22388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:07:44.472824 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41520 (local address 127.11.102.3:46621)
0620 06:07:44.474038 (+ 1214us) server_negotiation.cc:206] Beginning negotiation
0620 06:07:44.474046 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 06:07:44.474064 (+ 18us) server_negotiation.cc:407] Connection header received
0620 06:07:44.474119 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:07:44.474122 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:07:44.474274 (+ 152us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:07:44.474437 (+ 163us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 06:07:44.474776 (+ 339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:07:44.476114 (+ 1338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:07:44.477398 (+ 1284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:07:44.478226 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:07:44.478341 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:07:44.478482 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0620 06:07:44.480519 (+ 2037us) server_negotiation.cc:299] Negotiation successful
0620 06:07:44.480606 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1130,"thread_start_us":121,"threads_started":1}
I20260620 06:07:44.482218 22040 tablet_service.cc:1467] Processing CreateTablet for tablet 9b6caffb21b14e02a9fc688890d5b4b2 (DEFAULT_TABLE table=test-table [id=7d859cf75c04452b93a5d682c78cbe01]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 06:07:44.482544 22174 tablet_service.cc:1467] Processing CreateTablet for tablet 9b6caffb21b14e02a9fc688890d5b4b2 (DEFAULT_TABLE table=test-table [id=7d859cf75c04452b93a5d682c78cbe01]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 06:07:44.483386 22040 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9b6caffb21b14e02a9fc688890d5b4b2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 06:07:44.483525 22174 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9b6caffb21b14e02a9fc688890d5b4b2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 06:07:44.483347 22308 tablet_service.cc:1467] Processing CreateTablet for tablet 9b6caffb21b14e02a9fc688890d5b4b2 (DEFAULT_TABLE table=test-table [id=7d859cf75c04452b93a5d682c78cbe01]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 06:07:44.484262 22308 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9b6caffb21b14e02a9fc688890d5b4b2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 06:07:44.489344 22391 tablet_bootstrap.cc:492] T 9b6caffb21b14e02a9fc688890d5b4b2 P 20efde36d9dc44a6b8c09f28eb3fed1e: Bootstrap starting.
I20260620 06:07:44.490710 22393 tablet_bootstrap.cc:492] T 9b6caffb21b14e02a9fc688890d5b4b2 P a347ab2184e54b1ca341ca15bc069b9d: Bootstrap starting.
I20260620 06:07:44.491350 22392 tablet_bootstrap.cc:492] T 9b6caffb21b14e02a9fc688890d5b4b2 P 2cf36ed63d0f4a6e8c002166660f0867: Bootstrap starting.
I20260620 06:07:44.492141 22391 tablet_bootstrap.cc:654] T 9b6caffb21b14e02a9fc688890d5b4b2 P 20efde36d9dc44a6b8c09f28eb3fed1e: Neither blocks nor log segments found. Creating new log.
I20260620 06:07:44.492944 22393 tablet_bootstrap.cc:654] T 9b6caffb21b14e02a9fc688890d5b4b2 P a347ab2184e54b1ca341ca15bc069b9d: Neither blocks nor log segments found. Creating new log.
I20260620 06:07:44.493005 22391 log.cc:824] T 9b6caffb21b14e02a9fc688890d5b4b2 P 20efde36d9dc44a6b8c09f28eb3fed1e: Log is configured to *not* fsync() on all Append() calls
I20260620 06:07:44.493336 22392 tablet_bootstrap.cc:654] T 9b6caffb21b14e02a9fc688890d5b4b2 P 2cf36ed63d0f4a6e8c002166660f0867: Neither blocks nor log segments found. Creating new log.
I20260620 06:07:44.494179 22393 log.cc:824] T 9b6caffb21b14e02a9fc688890d5b4b2 P a347ab2184e54b1ca341ca15bc069b9d: Log is configured to *not* fsync() on all Append() calls
I20260620 06:07:44.494477 22392 log.cc:824] T 9b6caffb21b14e02a9fc688890d5b4b2 P 2cf36ed63d0f4a6e8c002166660f0867: Log is configured to *not* fsync() on all Append() calls
I20260620 06:07:44.495157 22391 tablet_bootstrap.cc:492] T 9b6caffb21b14e02a9fc688890d5b4b2 P 20efde36d9dc44a6b8c09f28eb3fed1e: No bootstrap required, opened a new log
I20260620 06:07:44.495422 22391 ts_tablet_manager.cc:1397] T 9b6caffb21b14e02a9fc688890d5b4b2 P 20efde36d9dc44a6b8c09f28eb3fed1e: Time spent bootstrapping tablet: real 0.006s user 0.006s sys 0.000s
I20260620 06:07:44.495994 22393 tablet_bootstrap.cc:492] T 9b6caffb21b14e02a9fc688890d5b4b2 P a347ab2184e54b1ca341ca15bc069b9d: No bootstrap required, opened a new log
I20260620 06:07:44.496191 22393 ts_tablet_manager.cc:1397] T 9b6caffb21b14e02a9fc688890d5b4b2 P a347ab2184e54b1ca341ca15bc069b9d: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260620 06:07:44.496481 22392 tablet_bootstrap.cc:492] T 9b6caffb21b14e02a9fc688890d5b4b2 P 2cf36ed63d0f4a6e8c002166660f0867: No bootstrap required, opened a new log
I20260620 06:07:44.496649 22392 ts_tablet_manager.cc:1397] T 9b6caffb21b14e02a9fc688890d5b4b2 P 2cf36ed63d0f4a6e8c002166660f0867: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.000s
I20260620 06:07:44.499281 22393 raft_consensus.cc:348] T 9b6caffb21b14e02a9fc688890d5b4b2 P a347ab2184e54b1ca341ca15bc069b9d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cf36ed63d0f4a6e8c002166660f0867" member_type: VOTER last_known_addr { host: "127.11.102.1" port: 39933 } } peers { permanent_uuid: "a347ab2184e54b1ca341ca15bc069b9d" member_type: VOTER last_known_addr { host: "127.11.102.3" port: 46621 } } peers { permanent_uuid: "20efde36d9dc44a6b8c09f28eb3fed1e" member_type: VOTER last_known_addr { host: "127.11.102.2" port: 39443 } }
I20260620 06:07:44.499305 22391 raft_consensus.cc:348] T 9b6caffb21b14e02a9fc688890d5b4b2 P 20efde36d9dc44a6b8c09f28eb3fed1e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cf36ed63d0f4a6e8c002166660f0867" member_type: VOTER last_known_addr { host: "127.11.102.1" port: 39933 } } peers { permanent_uuid: "a347ab2184e54b1ca341ca15bc069b9d" member_type: VOTER last_known_addr { host: "127.11.102.3" port: 46621 } } peers { permanent_uuid: "20efde36d9dc44a6b8c09f28eb3fed1e" member_type: VOTER last_known_addr { host: "127.11.102.2" port: 39443 } }
I20260620 06:07:44.499516 22393 raft_consensus.cc:374] T 9b6caffb21b14e02a9fc688890d5b4b2 P a347ab2184e54b1ca341ca15bc069b9d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 06:07:44.499540 22391 raft_consensus.cc:374] T 9b6caffb21b14e02a9fc688890d5b4b2 P 20efde36d9dc44a6b8c09f28eb3fed1e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 06:07:44.499585 22393 raft_consensus.cc:729] T 9b6caffb21b14e02a9fc688890d5b4b2 P a347ab2184e54b1ca341ca15bc069b9d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a347ab2184e54b1ca341ca15bc069b9d, State: Initialized, Role: FOLLOWER
I20260620 06:07:44.499898 22391 raft_consensus.cc:729] T 9b6caffb21b14e02a9fc688890d5b4b2 P 20efde36d9dc44a6b8c09f28eb3fed1e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 20efde36d9dc44a6b8c09f28eb3fed1e, State: Initialized, Role: FOLLOWER
I20260620 06:07:44.500142 22393 consensus_queue.cc:260] T 9b6caffb21b14e02a9fc688890d5b4b2 P a347ab2184e54b1ca341ca15bc069b9d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cf36ed63d0f4a6e8c002166660f0867" member_type: VOTER last_known_addr { host: "127.11.102.1" port: 39933 } } peers { permanent_uuid: "a347ab2184e54b1ca341ca15bc069b9d" member_type: VOTER last_known_addr { host: "127.11.102.3" port: 46621 } } peers { permanent_uuid: "20efde36d9dc44a6b8c09f28eb3fed1e" member_type: VOTER last_known_addr { host: "127.11.102.2" port: 39443 } }
I20260620 06:07:44.500345 22391 consensus_queue.cc:260] T 9b6caffb21b14e02a9fc688890d5b4b2 P 20efde36d9dc44a6b8c09f28eb3fed1e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cf36ed63d0f4a6e8c002166660f0867" member_type: VOTER last_known_addr { host: "127.11.102.1" port: 39933 } } peers { permanent_uuid: "a347ab2184e54b1ca341ca15bc069b9d" member_type: VOTER last_known_addr { host: "127.11.102.3" port: 46621 } } peers { permanent_uuid: "20efde36d9dc44a6b8c09f28eb3fed1e" member_type: VOTER last_known_addr { host: "127.11.102.2" port: 39443 } }
I20260620 06:07:44.500259 22392 raft_consensus.cc:348] T 9b6caffb21b14e02a9fc688890d5b4b2 P 2cf36ed63d0f4a6e8c002166660f0867 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cf36ed63d0f4a6e8c002166660f0867" member_type: VOTER last_known_addr { host: "127.11.102.1" port: 39933 } } peers { permanent_uuid: "a347ab2184e54b1ca341ca15bc069b9d" member_type: VOTER last_known_addr { host: "127.11.102.3" port: 46621 } } peers { permanent_uuid: "20efde36d9dc44a6b8c09f28eb3fed1e" member_type: VOTER last_known_addr { host: "127.11.102.2" port: 39443 } }
I20260620 06:07:44.500495 22392 raft_consensus.cc:374] T 9b6caffb21b14e02a9fc688890d5b4b2 P 2cf36ed63d0f4a6e8c002166660f0867 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 06:07:44.500578 22392 raft_consensus.cc:729] T 9b6caffb21b14e02a9fc688890d5b4b2 P 2cf36ed63d0f4a6e8c002166660f0867 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2cf36ed63d0f4a6e8c002166660f0867, State: Initialized, Role: FOLLOWER
I20260620 06:07:44.500820 22374 heartbeater.cc:503] Master 127.11.102.62:46701 was elected leader, sending a full tablet report...
I20260620 06:07:44.501031 22240 heartbeater.cc:503] Master 127.11.102.62:46701 was elected leader, sending a full tablet report...
I20260620 06:07:44.501190 22393 ts_tablet_manager.cc:1428] T 9b6caffb21b14e02a9fc688890d5b4b2 P a347ab2184e54b1ca341ca15bc069b9d: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260620 06:07:44.501382 22391 ts_tablet_manager.cc:1428] T 9b6caffb21b14e02a9fc688890d5b4b2 P 20efde36d9dc44a6b8c09f28eb3fed1e: Time spent starting tablet: real 0.006s user 0.000s sys 0.005s
I20260620 06:07:44.502152 22392 consensus_queue.cc:260] T 9b6caffb21b14e02a9fc688890d5b4b2 P 2cf36ed63d0f4a6e8c002166660f0867 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cf36ed63d0f4a6e8c002166660f0867" member_type: VOTER last_known_addr { host: "127.11.102.1" port: 39933 } } peers { permanent_uuid: "a347ab2184e54b1ca341ca15bc069b9d" member_type: VOTER last_known_addr { host: "127.11.102.3" port: 46621 } } peers { permanent_uuid: "20efde36d9dc44a6b8c09f28eb3fed1e" member_type: VOTER last_known_addr { host: "127.11.102.2" port: 39443 } }
I20260620 06:07:44.502748 22106 heartbeater.cc:503] Master 127.11.102.62:46701 was elected leader, sending a full tablet report...
I20260620 06:07:44.503054 22392 ts_tablet_manager.cc:1428] T 9b6caffb21b14e02a9fc688890d5b4b2 P 2cf36ed63d0f4a6e8c002166660f0867: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
W20260620 06:07:44.512843 22107 tablet.cc:2369] T 9b6caffb21b14e02a9fc688890d5b4b2 P 2cf36ed63d0f4a6e8c002166660f0867: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 06:07:44.562501 22399 raft_consensus.cc:482] T 9b6caffb21b14e02a9fc688890d5b4b2 P 2cf36ed63d0f4a6e8c002166660f0867 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 06:07:44.562757 22399 raft_consensus.cc:504] T 9b6caffb21b14e02a9fc688890d5b4b2 P 2cf36ed63d0f4a6e8c002166660f0867 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cf36ed63d0f4a6e8c002166660f0867" member_type: VOTER last_known_addr { host: "127.11.102.1" port: 39933 } } peers { permanent_uuid: "a347ab2184e54b1ca341ca15bc069b9d" member_type: VOTER last_known_addr { host: "127.11.102.3" port: 46621 } } peers { permanent_uuid: "20efde36d9dc44a6b8c09f28eb3fed1e" member_type: VOTER last_known_addr { host: "127.11.102.2" port: 39443 } }
I20260620 06:07:44.563931 22399 leader_election.cc:288] T 9b6caffb21b14e02a9fc688890d5b4b2 P 2cf36ed63d0f4a6e8c002166660f0867 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a347ab2184e54b1ca341ca15bc069b9d (127.11.102.3:46621), 20efde36d9dc44a6b8c09f28eb3fed1e (127.11.102.2:39443)
I20260620 06:07:44.569171 22388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:07:44.564344 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.102.1:40997 (local address 127.11.102.3:46621)
0620 06:07:44.564472 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0620 06:07:44.564477 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 06:07:44.564710 (+ 233us) server_negotiation.cc:407] Connection header received
0620 06:07:44.564865 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:07:44.564868 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:07:44.564925 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:07:44.565008 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 06:07:44.565397 (+ 389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:07:44.566190 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:07:44.567911 (+ 1721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:07:44.568517 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:07:44.568672 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:07:44.568712 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0620 06:07:44.568968 (+ 256us) server_negotiation.cc:299] Negotiation successful
0620 06:07:44.569039 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260620 06:07:44.569171 22390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:07:44.564347 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.102.1:50535 (local address 127.11.102.2:39443)
0620 06:07:44.564472 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0620 06:07:44.564477 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 06:07:44.564782 (+ 305us) server_negotiation.cc:407] Connection header received
0620 06:07:44.564880 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:07:44.564883 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:07:44.564929 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:07:44.565008 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 06:07:44.565397 (+ 389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:07:44.566179 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:07:44.567911 (+ 1732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:07:44.568521 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:07:44.568652 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:07:44.568706 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0620 06:07:44.568968 (+ 262us) server_negotiation.cc:299] Negotiation successful
0620 06:07:44.569039 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260620 06:07:44.569259 22400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:07:44.564213 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.102.3:46621 (local address 127.11.102.1:40997)
0620 06:07:44.564594 (+ 381us) negotiation.cc:107] Waiting for socket to connect
0620 06:07:44.564611 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 06:07:44.564744 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:07:44.565037 (+ 293us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:07:44.565040 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:07:44.565081 (+ 41us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 06:07:44.565279 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:07:44.565286 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:07:44.566304 (+ 1018us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:07:44.566308 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:07:44.567755 (+ 1447us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:07:44.567766 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:07:44.568652 (+ 886us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:07:44.568656 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:07:44.568797 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:07:44.568856 (+ 59us) client_negotiation.cc:769] Sending connection context
0620 06:07:44.568936 (+ 80us) client_negotiation.cc:240] Negotiation successful
0620 06:07:44.569073 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":278,"mutex_wait_us":175,"spinlock_wait_cycles":5120,"thread_start_us":109,"threads_started":1}
I20260620 06:07:44.569262 22401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:07:44.564213 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.102.2:39443 (local address 127.11.102.1:50535)
0620 06:07:44.564692 (+ 479us) negotiation.cc:107] Waiting for socket to connect
0620 06:07:44.564704 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 06:07:44.564785 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:07:44.565037 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:07:44.565040 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:07:44.565079 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 06:07:44.565280 (+ 201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:07:44.565286 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:07:44.566304 (+ 1018us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:07:44.566308 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:07:44.567755 (+ 1447us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:07:44.567766 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:07:44.568652 (+ 886us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:07:44.568656 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:07:44.568777 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:07:44.568856 (+ 79us) client_negotiation.cc:769] Sending connection context
0620 06:07:44.568936 (+ 80us) client_negotiation.cc:240] Negotiation successful
0620 06:07:44.569073 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":418,"mutex_wait_us":173,"spinlock_wait_cycles":29440,"thread_start_us":97,"threads_started":1}
I20260620 06:07:44.569921 22194 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b6caffb21b14e02a9fc688890d5b4b2" candidate_uuid: "2cf36ed63d0f4a6e8c002166660f0867" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "20efde36d9dc44a6b8c09f28eb3fed1e" is_pre_election: true
I20260620 06:07:44.569984 22328 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b6caffb21b14e02a9fc688890d5b4b2" candidate_uuid: "2cf36ed63d0f4a6e8c002166660f0867" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a347ab2184e54b1ca341ca15bc069b9d" is_pre_election: true
I20260620 06:07:44.570242 22194 raft_consensus.cc:2454] T 9b6caffb21b14e02a9fc688890d5b4b2 P 20efde36d9dc44a6b8c09f28eb3fed1e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2cf36ed63d0f4a6e8c002166660f0867 in term 0.
I20260620 06:07:44.570278 22328 raft_consensus.cc:2454] T 9b6caffb21b14e02a9fc688890d5b4b2 P a347ab2184e54b1ca341ca15bc069b9d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2cf36ed63d0f4a6e8c002166660f0867 in term 0.
I20260620 06:07:44.570734 21996 leader_election.cc:302] T 9b6caffb21b14e02a9fc688890d5b4b2 P 2cf36ed63d0f4a6e8c002166660f0867 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 20efde36d9dc44a6b8c09f28eb3fed1e, 2cf36ed63d0f4a6e8c002166660f0867, a347ab2184e54b1ca341ca15bc069b9d; no voters:
I20260620 06:07:44.570986 22399 raft_consensus.cc:2781] T 9b6caffb21b14e02a9fc688890d5b4b2 P 2cf36ed63d0f4a6e8c002166660f0867 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 06:07:44.571079 22399 raft_consensus.cc:482] T 9b6caffb21b14e02a9fc688890d5b4b2 P 2cf36ed63d0f4a6e8c002166660f0867 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 06:07:44.571173 22399 raft_consensus.cc:3037] T 9b6caffb21b14e02a9fc688890d5b4b2 P 2cf36ed63d0f4a6e8c002166660f0867 [term 0 FOLLOWER]: Advancing to term 1
I20260620 06:07:44.572515 22399 raft_consensus.cc:504] T 9b6caffb21b14e02a9fc688890d5b4b2 P 2cf36ed63d0f4a6e8c002166660f0867 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cf36ed63d0f4a6e8c002166660f0867" member_type: VOTER last_known_addr { host: "127.11.102.1" port: 39933 } } peers { permanent_uuid: "a347ab2184e54b1ca341ca15bc069b9d" member_type: VOTER last_known_addr { host: "127.11.102.3" port: 46621 } } peers { permanent_uuid: "20efde36d9dc44a6b8c09f28eb3fed1e" member_type: VOTER last_known_addr { host: "127.11.102.2" port: 39443 } }
I20260620 06:07:44.572894 22399 leader_election.cc:288] T 9b6caffb21b14e02a9fc688890d5b4b2 P 2cf36ed63d0f4a6e8c002166660f0867 [CANDIDATE]: Term 1 election: Requested vote from peers a347ab2184e54b1ca341ca15bc069b9d (127.11.102.3:46621), 20efde36d9dc44a6b8c09f28eb3fed1e (127.11.102.2:39443)
I20260620 06:07:44.573280 22194 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b6caffb21b14e02a9fc688890d5b4b2" candidate_uuid: "2cf36ed63d0f4a6e8c002166660f0867" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "20efde36d9dc44a6b8c09f28eb3fed1e"
I20260620 06:07:44.573293 22328 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b6caffb21b14e02a9fc688890d5b4b2" candidate_uuid: "2cf36ed63d0f4a6e8c002166660f0867" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a347ab2184e54b1ca341ca15bc069b9d"
I20260620 06:07:44.573426 22194 raft_consensus.cc:3037] T 9b6caffb21b14e02a9fc688890d5b4b2 P 20efde36d9dc44a6b8c09f28eb3fed1e [term 0 FOLLOWER]: Advancing to term 1
I20260620 06:07:44.573436 22328 raft_consensus.cc:3037] T 9b6caffb21b14e02a9fc688890d5b4b2 P a347ab2184e54b1ca341ca15bc069b9d [term 0 FOLLOWER]: Advancing to term 1
I20260620 06:07:44.574579 22328 raft_consensus.cc:2454] T 9b6caffb21b14e02a9fc688890d5b4b2 P a347ab2184e54b1ca341ca15bc069b9d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2cf36ed63d0f4a6e8c002166660f0867 in term 1.
I20260620 06:07:44.574779 22194 raft_consensus.cc:2454] T 9b6caffb21b14e02a9fc688890d5b4b2 P 20efde36d9dc44a6b8c09f28eb3fed1e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2cf36ed63d0f4a6e8c002166660f0867 in term 1.
I20260620 06:07:44.575093 21996 leader_election.cc:302] T 9b6caffb21b14e02a9fc688890d5b4b2 P 2cf36ed63d0f4a6e8c002166660f0867 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20efde36d9dc44a6b8c09f28eb3fed1e, 2cf36ed63d0f4a6e8c002166660f0867; no voters:
I20260620 06:07:44.575318 22399 raft_consensus.cc:2781] T 9b6caffb21b14e02a9fc688890d5b4b2 P 2cf36ed63d0f4a6e8c002166660f0867 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 06:07:44.575546 22399 raft_consensus.cc:686] T 9b6caffb21b14e02a9fc688890d5b4b2 P 2cf36ed63d0f4a6e8c002166660f0867 [term 1 LEADER]: Becoming Leader. State: Replica: 2cf36ed63d0f4a6e8c002166660f0867, State: Running, Role: LEADER
I20260620 06:07:44.575891 22399 consensus_queue.cc:237] T 9b6caffb21b14e02a9fc688890d5b4b2 P 2cf36ed63d0f4a6e8c002166660f0867 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cf36ed63d0f4a6e8c002166660f0867" member_type: VOTER last_known_addr { host: "127.11.102.1" port: 39933 } } peers { permanent_uuid: "a347ab2184e54b1ca341ca15bc069b9d" member_type: VOTER last_known_addr { host: "127.11.102.3" port: 46621 } } peers { permanent_uuid: "20efde36d9dc44a6b8c09f28eb3fed1e" member_type: VOTER last_known_addr { host: "127.11.102.2" port: 39443 } }
I20260620 06:07:44.579052 21915 catalog_manager.cc:5495] T 9b6caffb21b14e02a9fc688890d5b4b2 P 2cf36ed63d0f4a6e8c002166660f0867 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2cf36ed63d0f4a6e8c002166660f0867 (127.11.102.1). New cstate: current_term: 1 leader_uuid: "2cf36ed63d0f4a6e8c002166660f0867" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cf36ed63d0f4a6e8c002166660f0867" member_type: VOTER last_known_addr { host: "127.11.102.1" port: 39933 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a347ab2184e54b1ca341ca15bc069b9d" member_type: VOTER last_known_addr { host: "127.11.102.3" port: 46621 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "20efde36d9dc44a6b8c09f28eb3fed1e" member_type: VOTER last_known_addr { host: "127.11.102.2" port: 39443 } health_report { overall_health: UNKNOWN } } }
I20260620 06:07:44.604293 22387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:07:44.600627 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52292 (local address 127.11.102.1:39933)
0620 06:07:44.600755 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0620 06:07:44.600759 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 06:07:44.600772 (+ 13us) server_negotiation.cc:407] Connection header received
0620 06:07:44.600841 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:07:44.600843 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:07:44.600892 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:07:44.600987 (+ 95us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 06:07:44.601310 (+ 323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:07:44.601878 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:07:44.602762 (+ 884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:07:44.602985 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:07:44.603073 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:07:44.603434 (+ 361us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 06:07:44.603944 (+ 510us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 06:07:44.604041 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0620 06:07:44.604135 (+ 94us) server_negotiation.cc:299] Negotiation successful
0620 06:07:44.604187 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260620 06:07:44.610245 22194 raft_consensus.cc:1261] T 9b6caffb21b14e02a9fc688890d5b4b2 P 20efde36d9dc44a6b8c09f28eb3fed1e [term 1 FOLLOWER]: Refusing update from remote peer 2cf36ed63d0f4a6e8c002166660f0867: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 06:07:44.610266 22328 raft_consensus.cc:1261] T 9b6caffb21b14e02a9fc688890d5b4b2 P a347ab2184e54b1ca341ca15bc069b9d [term 1 FOLLOWER]: Refusing update from remote peer 2cf36ed63d0f4a6e8c002166660f0867: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 06:07:44.610970 22402 consensus_queue.cc:1035] T 9b6caffb21b14e02a9fc688890d5b4b2 P 2cf36ed63d0f4a6e8c002166660f0867 [LEADER]: Connected to new peer: Peer: permanent_uuid: "20efde36d9dc44a6b8c09f28eb3fed1e" member_type: VOTER last_known_addr { host: "127.11.102.2" port: 39443 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 06:07:44.611155 22399 consensus_queue.cc:1035] T 9b6caffb21b14e02a9fc688890d5b4b2 P 2cf36ed63d0f4a6e8c002166660f0867 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a347ab2184e54b1ca341ca15bc069b9d" member_type: VOTER last_known_addr { host: "127.11.102.3" port: 46621 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 06:07:44.618257 22407 mvcc.cc:205] Tried to move back new op lower bound from 7298808482236121088 to 7298808482105520128. Current Snapshot: MvccSnapshot[applied={T|T < 7298808482236121088}]
I20260620 06:07:44.618383 22405 mvcc.cc:205] Tried to move back new op lower bound from 7298808482236121088 to 7298808482105520128. Current Snapshot: MvccSnapshot[applied={T|T < 7298808482236121088}]
I20260620 06:07:44.619999 22406 mvcc.cc:205] Tried to move back new op lower bound from 7298808482236121088 to 7298808482105520128. Current Snapshot: MvccSnapshot[applied={T|T < 7298808482236121088}]
I20260620 06:07:44.625959 21915 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56492:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 06:07:44.626091 21915 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56492:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 06:07:44.628363 21915 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b71d23ee5a044402b769ed9b36de8a06: Sending DeleteTablet for 3 replicas of tablet 9b6caffb21b14e02a9fc688890d5b4b2
I20260620 06:07:44.629067 22308 tablet_service.cc:1514] Processing DeleteTablet for tablet 9b6caffb21b14e02a9fc688890d5b4b2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 06:07:44 UTC) from {username='kudu', principal='kudu/127.11.102.62@KRBTEST.COM'} at 127.0.0.1:41520
I20260620 06:07:44.629067 22040 tablet_service.cc:1514] Processing DeleteTablet for tablet 9b6caffb21b14e02a9fc688890d5b4b2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 06:07:44 UTC) from {username='kudu', principal='kudu/127.11.102.62@KRBTEST.COM'} at 127.0.0.1:52276
I20260620 06:07:44.629217 22174 tablet_service.cc:1514] Processing DeleteTablet for tablet 9b6caffb21b14e02a9fc688890d5b4b2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 06:07:44 UTC) from {username='kudu', principal='kudu/127.11.102.62@KRBTEST.COM'} at 127.0.0.1:43012
I20260620 06:07:44.629554 22414 tablet_replica.cc:330] T 9b6caffb21b14e02a9fc688890d5b4b2 P 2cf36ed63d0f4a6e8c002166660f0867: stopping tablet replica
I20260620 06:07:44.629670 22415 tablet_replica.cc:330] T 9b6caffb21b14e02a9fc688890d5b4b2 P a347ab2184e54b1ca341ca15bc069b9d: stopping tablet replica
I20260620 06:07:44.629909 22414 raft_consensus.cc:2229] T 9b6caffb21b14e02a9fc688890d5b4b2 P 2cf36ed63d0f4a6e8c002166660f0867 [term 1 LEADER]: Raft consensus shutting down.
I20260620 06:07:44.629938 22415 raft_consensus.cc:2229] T 9b6caffb21b14e02a9fc688890d5b4b2 P a347ab2184e54b1ca341ca15bc069b9d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 06:07:44.630242 22415 raft_consensus.cc:2258] T 9b6caffb21b14e02a9fc688890d5b4b2 P a347ab2184e54b1ca341ca15bc069b9d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 06:07:44.630334 11672 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9HMp43/build/debug/bin/kudu with pid 21977
I20260620 06:07:44.632212 22416 tablet_replica.cc:330] T 9b6caffb21b14e02a9fc688890d5b4b2 P 20efde36d9dc44a6b8c09f28eb3fed1e: stopping tablet replica
I20260620 06:07:44.632489 22416 raft_consensus.cc:2229] T 9b6caffb21b14e02a9fc688890d5b4b2 P 20efde36d9dc44a6b8c09f28eb3fed1e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 06:07:44.632782 22416 raft_consensus.cc:2258] T 9b6caffb21b14e02a9fc688890d5b4b2 P 20efde36d9dc44a6b8c09f28eb3fed1e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 06:07:44.632849 22415 ts_tablet_manager.cc:1905] T 9b6caffb21b14e02a9fc688890d5b4b2 P a347ab2184e54b1ca341ca15bc069b9d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 06:07:44.634135 22416 ts_tablet_manager.cc:1905] T 9b6caffb21b14e02a9fc688890d5b4b2 P 20efde36d9dc44a6b8c09f28eb3fed1e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 06:07:44.635655 22415 ts_tablet_manager.cc:1918] T 9b6caffb21b14e02a9fc688890d5b4b2 P a347ab2184e54b1ca341ca15bc069b9d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 06:07:44.635751 22415 log.cc:1196] T 9b6caffb21b14e02a9fc688890d5b4b2 P a347ab2184e54b1ca341ca15bc069b9d: Deleting WAL directory at /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781935603120653-11672-0/minicluster-data/ts-2/wal/wals/9b6caffb21b14e02a9fc688890d5b4b2
I20260620 06:07:44.636075 22415 ts_tablet_manager.cc:1939] T 9b6caffb21b14e02a9fc688890d5b4b2 P a347ab2184e54b1ca341ca15bc069b9d: Deleting consensus metadata
I20260620 06:07:44.636796 22416 ts_tablet_manager.cc:1918] T 9b6caffb21b14e02a9fc688890d5b4b2 P 20efde36d9dc44a6b8c09f28eb3fed1e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 06:07:44.636881 22416 log.cc:1196] T 9b6caffb21b14e02a9fc688890d5b4b2 P 20efde36d9dc44a6b8c09f28eb3fed1e: Deleting WAL directory at /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781935603120653-11672-0/minicluster-data/ts-1/wal/wals/9b6caffb21b14e02a9fc688890d5b4b2
I20260620 06:07:44.637179 22416 ts_tablet_manager.cc:1939] T 9b6caffb21b14e02a9fc688890d5b4b2 P 20efde36d9dc44a6b8c09f28eb3fed1e: Deleting consensus metadata
I20260620 06:07:44.637300 21899 catalog_manager.cc:4841] TS a347ab2184e54b1ca341ca15bc069b9d (127.11.102.3:46621): tablet 9b6caffb21b14e02a9fc688890d5b4b2 (table test-table [id=7d859cf75c04452b93a5d682c78cbe01]) successfully deleted
W20260620 06:07:44.637348 21900 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.11.102.1:39933 (error 108)
I20260620 06:07:44.638032 11672 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9HMp43/build/debug/bin/kudu with pid 22111
I20260620 06:07:44.638322 22389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:07:44.638030 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.102.1:39933 (local address 127.0.0.1:52302)
0620 06:07:44.638155 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0620 06:07:44.638231 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.102.1:39933: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260620 06:07:44.638687 21900 catalog_manager.cc:4584] TS 2cf36ed63d0f4a6e8c002166660f0867 (127.11.102.1:39933): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 9b6caffb21b14e02a9fc688890d5b4b2: Network error: Client connection negotiation failed: client connection to 127.11.102.1:39933: connect: Connection refused (error 111)
I20260620 06:07:44.644315 11672 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9HMp43/build/debug/bin/kudu with pid 22245
I20260620 06:07:44.644883 22389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:07:44.644505 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.102.2:39443 (local address 127.0.0.1:43028)
0620 06:07:44.644720 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0620 06:07:44.644777 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.102.2:39443: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107}
I20260620 06:07:44.650067 11672 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9HMp43/build/debug/bin/kudu with pid 21883
2026-06-20T06:07:44Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (2224 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781935603120653-11672-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 06:07:44 dist-test-slave-9hm5 krb5kdc[22420](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 06:07:44 dist-test-slave-9hm5 krb5kdc[22420](info): set up 2 sockets
Jun 20 06:07:44 dist-test-slave-9hm5 krb5kdc[22420](info): commencing operation
krb5kdc: starting...
W20260620 06:07:45.696671 11672 mini_kdc.cc:121] Time spent starting KDC: real 1.025s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781935603120653-11672-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781935603120653-11672-0/krb5kdc/test-user.keytab.
Jun 20 06:07:45 dist-test-slave-9hm5 krb5kdc[22420](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935665, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T06:07:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T06:07:45Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.11.102.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.11.102.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.11.102.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781935603120653-11672-0/krb5kdc/rangeradmin_127.11.102.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.11.102.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781935603120653-11672-0/krb5kdc/rangeradmin_127.11.102.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.11.102.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.11.102.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.11.102.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781935603120653-11672-0/krb5kdc/rangerlookup_127.11.102.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.11.102.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781935603120653-11672-0/krb5kdc/rangerlookup_127.11.102.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.11.102.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.11.102.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.11.102.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781935603120653-11672-0/krb5kdc/HTTP_127.11.102.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.11.102.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781935603120653-11672-0/krb5kdc/HTTP_127.11.102.20@KRBTEST.COM.keytab.
I20260620 06:07:45.860460 11672 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781935603120653-11672-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-task9HMp43/build/debug/bin/postgres/pg_ctl -D /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781935603120653-11672-0/minicluster-data/postgres -l logfile start
2026-06-20 06:07:48.287 UTC [22450] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-06-20 06:07:48.287 UTC [22450] LOG: listening on IPv4 address "127.11.102.20", port 56041
2026-06-20 06:07:48.291 UTC [22450] LOG: listening on Unix socket "/tmp/.s.PGSQL.56041"
2026-06-20 06:07:48.326 UTC [22453] LOG: database system was shut down at 2026-06-20 06:07:46 UTC
2026-06-20 06:07:48.330 UTC [22450] LOG: database system is ready to accept connections
I20260620 06:07:49.284493 11672 mini_postgres.cc:96] Postgres bound to 56041
2026-06-20 06:07:49.290 UTC [22461] FATAL: database "slave" does not exist
127.11.102.20:56041 - accepting connections
I20260620 06:07:49.291365 11672 mini_ranger.cc:161] Starting Ranger...
I20260620 06:07:49.306234 11672 mini_ranger.cc:84] Created miniranger Postgres user
I20260620 06:07:49.755813 11672 mini_ranger.cc:87] Created ranger Postgres database
I20260620 06:07:49.755914 11672 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781935603120653-11672-0/minicluster-data/ranger-admin
2026-06-20 06:07:50,052 [I] DB FLAVOR :POSTGRES
2026-06-20 06:07:50,053 [I] --------- Verifying Ranger DB connection ---------
2026-06-20 06:07:50,053 [I] Checking connection..
2026-06-20 06:07:50,053 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task9HMp43/build/debug/bin/postgresql.jar:/tmp/dist-test-task9HMp43/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.102.20:56041/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-06-20 06:07:50,299 [I] Checking connection passed.
2026-06-20 06:07:50,299 [I] --------- Verifying version history table ---------
2026-06-20 06:07:50,299 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task9HMp43/build/debug/bin/postgresql.jar:/tmp/dist-test-task9HMp43/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.102.20:56041/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-20 06:07:50,526 [I] Table x_db_version_h does not exist in database ranger
2026-06-20 06:07:50,527 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task9HMp43/build/debug/bin/postgresql.jar:/tmp/dist-test-task9HMp43/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.102.20:56041/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-20 06:07:50,748 [I] Table x_db_version_h does not exist in database ranger
2026-06-20 06:07:50,748 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-06-20 06:07:50,748 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task9HMp43/build/debug/bin/postgresql.jar:/tmp/dist-test-task9HMp43/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.102.20:56041/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task9HMp43/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-06-20 06:07:50.972 UTC [22525] WARNING: there is no transaction in progress
2026-06-20 06:07:50,978 [I] create_dbversion_catalog.sql file imported successfully
2026-06-20 06:07:50,978 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task9HMp43/build/debug/bin/postgresql.jar:/tmp/dist-test-task9HMp43/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.102.20:56041/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-20 06:07:51,212 [I] Table x_db_version_h already exists in database 'ranger'
2026-06-20 06:07:51,212 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-20 06:07:51,212 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task9HMp43/build/debug/bin/postgresql.jar:/tmp/dist-test-task9HMp43/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.102.20:56041/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-06-20 06:07:51,487 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task9HMp43/build/debug/bin/postgresql.jar:/tmp/dist-test-task9HMp43/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.102.20:56041/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-06-20 06:07:51,728 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task9HMp43/build/debug/bin/postgresql.jar:/tmp/dist-test-task9HMp43/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.102.20:56041/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-9hm5.c.gcp-upstream.internal','N') ;"
2026-06-20 06:07:51,977 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task9HMp43/build/debug/bin/postgresql.jar:/tmp/dist-test-task9HMp43/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.102.20:56041/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-20 06:07:52,208 [I] Table x_portal_user does not exist in database ranger
2026-06-20 06:07:52,208 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task9HMp43/build/debug/bin/postgresql.jar:/tmp/dist-test-task9HMp43/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.102.20:56041/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-20 06:07:52,452 [I] Table x_policy_ref_group does not exist in database ranger
2026-06-20 06:07:52,452 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task9HMp43/build/debug/bin/postgresql.jar:/tmp/dist-test-task9HMp43/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.102.20:56041/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-20 06:07:52,691 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task9HMp43/build/debug/bin/postgresql.jar:/tmp/dist-test-task9HMp43/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.102.20:56041/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-20 06:07:52,924 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-06-20 06:07:52,924 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task9HMp43/build/debug/bin/postgresql.jar:/tmp/dist-test-task9HMp43/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.102.20:56041/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task9HMp43/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-06-20 06:07:53.874 UTC [22633] WARNING: there is no transaction in progress
2026-06-20 06:07:53.890 UTC [22633] WARNING: there is no transaction in progress
2026-06-20 06:07:53.905 UTC [22633] WARNING: there is no transaction in progress
2026-06-20 06:07:53.919 UTC [22633] WARNING: there is no transaction in progress
2026-06-20 06:07:53.937 UTC [22633] WARNING: there is no transaction in progress
2026-06-20 06:07:53.953 UTC [22633] WARNING: there is no transaction in progress
2026-06-20 06:07:54.016 UTC [22633] WARNING: there is no transaction in progress
2026-06-20 06:07:54.022 UTC [22633] WARNING: there is no transaction in progress
2026-06-20 06:07:54.036 UTC [22633] WARNING: there is no transaction in progress
2026-06-20 06:07:54.045 UTC [22633] WARNING: there is no transaction in progress
2026-06-20 06:07:54.056 UTC [22633] WARNING: there is no transaction in progress
2026-06-20 06:07:54.069 UTC [22633] WARNING: there is no transaction in progress
2026-06-20 06:07:54.079 UTC [22633] WARNING: there is no transaction in progress
2026-06-20 06:07:54.086 UTC [22633] WARNING: there is no transaction in progress
2026-06-20 06:07:54.092 UTC [22633] WARNING: there is no transaction in progress
2026-06-20 06:07:54.098 UTC [22633] WARNING: there is no transaction in progress
2026-06-20 06:07:54,708 [I] ranger_core_db_postgres.sql file imported successfully
2026-06-20 06:07:54,708 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task9HMp43/build/debug/bin/postgresql.jar:/tmp/dist-test-task9HMp43/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.102.20:56041/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-06-20 06:07:54,955 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-06-20 06:07:54,955 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task9HMp43/build/debug/bin/postgresql.jar:/tmp/dist-test-task9HMp43/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.102.20:56041/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-20 06:07:55,187 [I] Table x_portal_user already exists in database 'ranger'
2026-06-20 06:07:55,187 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task9HMp43/build/debug/bin/postgresql.jar:/tmp/dist-test-task9HMp43/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.102.20:56041/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-20 06:07:55,431 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-06-20 06:07:55,431 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task9HMp43/build/debug/bin/postgresql.jar:/tmp/dist-test-task9HMp43/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.102.20:56041/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-20 06:07:55,655 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task9HMp43/build/debug/bin/postgresql.jar:/tmp/dist-test-task9HMp43/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.102.20:56041/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-20 06:07:55,880 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task9HMp43/build/debug/bin/postgresql.jar:/tmp/dist-test-task9HMp43/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.102.20:56041/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-9hm5.c.gcp-upstream.internal';"
2026-06-20 06:07:56,120 [I] CORE_DB_SCHEMA import status has been updated
2026-06-20 06:07:56,120 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task9HMp43/build/debug/bin/postgresql.jar:/tmp/dist-test-task9HMp43/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.102.20:56041/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-06-20 06:07:56,372 [I] DB_PATCHES have already been applied
I20260620 06:07:56.377707 11672 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781935603120653-11672-0/minicluster-data/ranger-admin:/tmp/dist-test-task9HMp43/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-task9HMp43/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-task9HMp43/build/debug/bin/postgresql.jar:/tmp/dist-test-task9HMp43/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260620 06:07:56.377769 11672 mini_ranger.cc:193] Using host: 127.11.102.20
I20260620 06:07:56.381099 11672 mini_ranger.cc:227] Ranger admin URL: http://127.11.102.20:35351
Jun 20, 2026 6:07:56 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 20, 2026 6:07:56 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 20, 2026 6:07:56 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781935603120653-11672-0/minicluster-data/ranger-admin/ews/webapp
Jun 20, 2026 6:07:56 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781935603120653-11672-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Jun 20, 2026 6:07:56 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781935603120653-11672-0/minicluster-data/ranger-admin/ews/webapp] .....
Jun 20, 2026 6:07:57 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781935603120653-11672-0/minicluster-data/ranger-admin/ews/webapp].
Jun 20, 2026 6:07:57 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260620 06:07:57.470871 11672 mini_ranger.cc:160] Time spent starting Ranger: real 8.180s user 0.007s sys 0.002s
Jun 20, 2026 6:08:17 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Jun 20, 2026 6:08:17 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Jun 20, 2026 6:08:17 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Jun 20, 2026 6:08:17 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Jun 20, 2026 6:08:17 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Jun 20, 2026 6:08:17 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Jun 20, 2026 6:08:17 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Jun 20, 2026 6:08:17 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Jun 20, 2026 6:08:17 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Jun 20, 2026 6:08:17 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Jun 20, 2026 6:08:17 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Jun 20, 2026 6:08:17 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Jun 20, 2026 6:08:17 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Jun 20, 2026 6:08:17 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Jun 20, 2026 6:08:17 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Jun 20, 2026 6:08:17 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Jun 20, 2026 6:08:17 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Jun 20, 2026 6:08:18 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260620 06:08:19.364151 11672 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.11.102.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.11.102.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.11.102.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781935603120653-11672-0/krb5kdc/rangerkms_127.11.102.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.11.102.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781935603120653-11672-0/krb5kdc/rangerkms_127.11.102.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.11.102.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.11.102.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.11.102.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781935603120653-11672-0/krb5kdc/HTTP_127.11.102.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.11.102.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781935603120653-11672-0/krb5kdc/HTTP_127.11.102.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Jun 20 06:08:19 dist-test-slave-9hm5 krb5kdc[22420](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935699, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260620 06:08:19.446677 11672 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260620 06:08:19.455667 11672 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260620 06:08:20.279784 11672 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260620 06:08:20.279886 11672 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781935603120653-11672-0/minicluster-data/ranger-kms
I20260620 06:08:20.279987 11672 mini_ranger_kms.cc:227] Using postgres at 127.11.102.20:56041
2026-06-20 06:08:20,464 [I] DB FLAVOR :POSTGRES
2026-06-20 06:08:20,464 [I] --------- Verifying Ranger DB connection ---------
2026-06-20 06:08:20,464 [I] Checking connection
2026-06-20 06:08:20,464 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task9HMp43/build/debug/bin/postgresql.jar:/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781935603120653-11672-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.102.20:56041/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-06-20 06:08:20,709 [I] connection success
2026-06-20 06:08:20,709 [I] --------- Verifying Ranger DB tables ---------
2026-06-20 06:08:20,709 [I] Verifying table ranger_masterkey in database rangerkms
2026-06-20 06:08:20,709 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task9HMp43/build/debug/bin/postgresql.jar:/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781935603120653-11672-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.102.20:56041/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-06-20 06:08:20,950 [I] Table ranger_masterkey does not exist in database rangerkms
2026-06-20 06:08:20,950 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-20 06:08:20,950 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-06-20 06:08:20,950 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task9HMp43/build/debug/bin/postgresql.jar:/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781935603120653-11672-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.102.20:56041/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781935603120653-11672-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-06-20 06:08:21,207 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260620 06:08:21.686875 11672 mini_ranger_kms.cc:310] Created kms service
I20260620 06:08:21.891355 11672 mini_ranger_kms.cc:326] Created kudu user
I20260620 06:08:21.936172 11672 mini_ranger_kms.cc:343] Created rangerkms user
Jun 20 06:08:21 dist-test-slave-9hm5 krb5kdc[22420](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935699, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.11.102.20@KRBTEST.COM
I20260620 06:08:22.320199 11672 mini_ranger_kms.cc:384] Added ranger policy
I20260620 06:08:22.320374 11672 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781935603120653-11672-0/minicluster-data/ranger-kms:/tmp/dist-test-task9HMp43/build/debug/bin/postgresql.jar:/tmp/dist-test-task9HMp43/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-task9HMp43/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260620 06:08:22.320398 11672 mini_ranger_kms.cc:240] Using host: 127.11.102.20
I20260620 06:08:22.324241 11672 mini_ranger_kms.cc:276] Ranger KMS PID: 22813
I20260620 06:08:22.324406 11672 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.11.102.20:40509
Jun 20, 2026 6:08:23 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 20, 2026 6:08:23 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 20, 2026 6:08:23 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781935603120653-11672-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Jun 20, 2026 6:08:23 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781935603120653-11672-0/minicluster-data/ranger-kms/ews/webapp] .....
Jun 20, 2026 6:08:23 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781935603120653-11672-0/minicluster-data/ranger-kms/ews/webapp].
Jun 20, 2026 6:08:23 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260620 06:08:24.541704 11672 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.095s user 0.000s sys 0.168s
I20260620 06:08:24.541839 11672 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260620 06:08:24.541941 11672 mini_ranger_kms.cc:401] 127.11.102.20:40509/kms/v1/keys
Jun 20 06:08:24 dist-test-slave-9hm5 krb5kdc[22420](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781935704, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.11.102.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Jun 20 06:08:25 dist-test-slave-9hm5 krb5kdc[22420](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935704, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.11.102.20@KRBTEST.COM for HTTP/127.11.102.20@KRBTEST.COM
Jun 20 06:08:25 dist-test-slave-9hm5 krb5kdc[22420](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935704, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.11.102.20@KRBTEST.COM for HTTP/127.11.102.20@KRBTEST.COM
Jun 20 06:08:27 dist-test-slave-9hm5 krb5kdc[22420](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935707, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.11.102.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.102.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.102.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781935603120653-11672-0/krb5kdc/kudu_127.11.102.62.keytab.
Entry for principal kudu/127.11.102.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781935603120653-11672-0/krb5kdc/kudu_127.11.102.62.keytab.
I20260620 06:08:27.694417 11672 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9HMp43/build/debug/bin/kudu
/tmp/dist-test-task9HMp43/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781935603120653-11672-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781935603120653-11672-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781935603120653-11672-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781935603120653-11672-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.102.62:37325
--webserver_interface=127.11.102.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781935603120653-11672-0/krb5kdc/kudu_127.11.102.62.keytab
--principal=kudu/127.11.102.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.102.20:35851
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.102.62:37325
--ranger_config_path=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781935603120653-11672-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.11.102.20:40509/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781935603120653-11672-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781935603120653-11672-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781935603120653-11672-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 06:08:27.809900 22864 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:08:27.810155 22864 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:08:27.810194 22864 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:08:27.814354 22864 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 06:08:27.814414 22864 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:08:27.814433 22864 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:08:27.814450 22864 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 06:08:27.814487 22864 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 06:08:27.819897 22864 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.102.20:35851
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781935603120653-11672-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781935603120653-11672-0/minicluster-data/master-0/wal
--ranger_kms_url=127.11.102.20:40509/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.11.102.62:37325
--ranger_config_path=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781935603120653-11672-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781935603120653-11672-0/krb5kdc/kudu_127.11.102.62.keytab
--principal=kudu/127.11.102.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.102.62:37325
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781935603120653-11672-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.102.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22864
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781935603120653-11672-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 05:28:48 UTC on ccycloud.quasar-fixhms.root.comops.site
build id 4457020
I20260620 06:08:27.820912 22864 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:08:27.821918 22864 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 06:08:27.830626 22864 server_base.cc:1034] running on GCE node
W20260620 06:08:27.830780 22870 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:08:27.830756 22869 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:08:27.830837 22872 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:08:27.831533 22864 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:08:27.832707 22864 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:08:27.833868 22864 hybrid_clock.cc:648] HybridClock initialized: now 1781935707833853 us; error 29 us; skew 500 ppm
Jun 20 06:08:27 dist-test-slave-9hm5 krb5kdc[22420](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935707, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.102.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 06:08:27.839787 22864 init.cc:377] Logged in from keytab as kudu/127.11.102.62@KRBTEST.COM (short username kudu)
I20260620 06:08:27.840973 22864 webserver.cc:460] Webserver started at http://127.11.102.62:40705/ using document root <none> and password file <none>
I20260620 06:08:27.841691 22864 fs_manager.cc:359] Metadata directory not provided
I20260620 06:08:27.841769 22864 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:08:27.842000 22864 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 06:08:27 dist-test-slave-9hm5 krb5kdc[22420](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935707, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.102.62@KRBTEST.COM for HTTP/127.11.102.20@KRBTEST.COM
I20260620 06:08:27.898731 22864 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781935603120653-11672-0/minicluster-data/master-0/data/instance:
uuid: "d7bfab7a8cf74c539a5ada4fe744c741"
format_stamp: "Formatted at 2026-06-20 06:08:27 on dist-test-slave-9hm5"
server_key: "a03212d3b7e85567669d2b0846a15a49"
server_key_iv: "960d43671560073a3783fda1d8ee9d67"
server_key_version: "kuduclusterkey@0"
I20260620 06:08:27.899458 22864 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781935603120653-11672-0/minicluster-data/master-0/wal/instance:
uuid: "d7bfab7a8cf74c539a5ada4fe744c741"
format_stamp: "Formatted at 2026-06-20 06:08:27 on dist-test-slave-9hm5"
server_key: "a03212d3b7e85567669d2b0846a15a49"
server_key_iv: "960d43671560073a3783fda1d8ee9d67"
server_key_version: "kuduclusterkey@0"
I20260620 06:08:27.904448 22864 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260620 06:08:27.922050 22879 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:08:27.923725 22864 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 06:08:27.923868 22864 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781935603120653-11672-0/minicluster-data/master-0/data,/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781935603120653-11672-0/minicluster-data/master-0/wal
uuid: "d7bfab7a8cf74c539a5ada4fe744c741"
format_stamp: "Formatted at 2026-06-20 06:08:27 on dist-test-slave-9hm5"
server_key: "a03212d3b7e85567669d2b0846a15a49"
server_key_iv: "960d43671560073a3783fda1d8ee9d67"
server_key_version: "kuduclusterkey@0"
I20260620 06:08:27.924008 22864 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781935603120653-11672-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781935603120653-11672-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781935603120653-11672-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:08:27.961658 22864 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:08:27.964864 22864 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:08:27.965068 22864 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:08:27.973097 22864 rpc_server.cc:307] RPC server started. Bound to: 127.11.102.62:37325
I20260620 06:08:27.973114 22931 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.102.62:37325 every 8 connection(s)
I20260620 06:08:27.974076 22864 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781935603120653-11672-0/minicluster-data/master-0/data/info.pb
I20260620 06:08:27.976833 22932 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 06:08:27.982209 22932 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d7bfab7a8cf74c539a5ada4fe744c741: Bootstrap starting.
I20260620 06:08:27.982759 11672 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9HMp43/build/debug/bin/kudu as pid 22864
I20260620 06:08:27.982892 11672 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781935603120653-11672-0/minicluster-data/master-0/wal/instance
I20260620 06:08:27.983173 11672 external_mini_cluster.cc:1428] Setting key 8a1838f99dc27f4d4cb701226c8b7063
I20260620 06:08:27.985193 22932 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d7bfab7a8cf74c539a5ada4fe744c741: Neither blocks nor log segments found. Creating new log.
I20260620 06:08:27.985934 22932 log.cc:824] T 00000000000000000000000000000000 P d7bfab7a8cf74c539a5ada4fe744c741: Log is configured to *not* fsync() on all Append() calls
I20260620 06:08:27.988193 22932 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d7bfab7a8cf74c539a5ada4fe744c741: No bootstrap required, opened a new log
Jun 20 06:08:27 dist-test-slave-9hm5 krb5kdc[22420](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935707, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.102.62@KRBTEST.COM
I20260620 06:08:27.992817 22932 raft_consensus.cc:348] T 00000000000000000000000000000000 P d7bfab7a8cf74c539a5ada4fe744c741 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d7bfab7a8cf74c539a5ada4fe744c741" member_type: VOTER last_known_addr { host: "127.11.102.62" port: 37325 } }
I20260620 06:08:27.993086 22932 raft_consensus.cc:374] T 00000000000000000000000000000000 P d7bfab7a8cf74c539a5ada4fe744c741 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 06:08:27.993148 22932 raft_consensus.cc:729] T 00000000000000000000000000000000 P d7bfab7a8cf74c539a5ada4fe744c741 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d7bfab7a8cf74c539a5ada4fe744c741, State: Initialized, Role: FOLLOWER
I20260620 06:08:27.993669 22932 consensus_queue.cc:260] T 00000000000000000000000000000000 P d7bfab7a8cf74c539a5ada4fe744c741 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d7bfab7a8cf74c539a5ada4fe744c741" member_type: VOTER last_known_addr { host: "127.11.102.62" port: 37325 } }
I20260620 06:08:27.993841 22932 raft_consensus.cc:388] T 00000000000000000000000000000000 P d7bfab7a8cf74c539a5ada4fe744c741 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 06:08:27.993916 22932 raft_consensus.cc:482] T 00000000000000000000000000000000 P d7bfab7a8cf74c539a5ada4fe744c741 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 06:08:27.994071 22932 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d7bfab7a8cf74c539a5ada4fe744c741 [term 0 FOLLOWER]: Advancing to term 1
I20260620 06:08:27.995314 22932 raft_consensus.cc:504] T 00000000000000000000000000000000 P d7bfab7a8cf74c539a5ada4fe744c741 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d7bfab7a8cf74c539a5ada4fe744c741" member_type: VOTER last_known_addr { host: "127.11.102.62" port: 37325 } }
I20260620 06:08:27.995723 22932 leader_election.cc:302] T 00000000000000000000000000000000 P d7bfab7a8cf74c539a5ada4fe744c741 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d7bfab7a8cf74c539a5ada4fe744c741; no voters:
I20260620 06:08:27.996100 22932 leader_election.cc:288] T 00000000000000000000000000000000 P d7bfab7a8cf74c539a5ada4fe744c741 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 06:08:27.996171 22937 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d7bfab7a8cf74c539a5ada4fe744c741 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 06:08:27.996374 22937 raft_consensus.cc:686] T 00000000000000000000000000000000 P d7bfab7a8cf74c539a5ada4fe744c741 [term 1 LEADER]: Becoming Leader. State: Replica: d7bfab7a8cf74c539a5ada4fe744c741, State: Running, Role: LEADER
I20260620 06:08:27.996680 22937 consensus_queue.cc:237] T 00000000000000000000000000000000 P d7bfab7a8cf74c539a5ada4fe744c741 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d7bfab7a8cf74c539a5ada4fe744c741" member_type: VOTER last_known_addr { host: "127.11.102.62" port: 37325 } }
I20260620 06:08:27.997418 22932 sys_catalog.cc:564] T 00000000000000000000000000000000 P d7bfab7a8cf74c539a5ada4fe744c741 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 06:08:27.998245 22938 sys_catalog.cc:455] T 00000000000000000000000000000000 P d7bfab7a8cf74c539a5ada4fe744c741 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d7bfab7a8cf74c539a5ada4fe744c741. Latest consensus state: current_term: 1 leader_uuid: "d7bfab7a8cf74c539a5ada4fe744c741" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d7bfab7a8cf74c539a5ada4fe744c741" member_type: VOTER last_known_addr { host: "127.11.102.62" port: 37325 } } }
I20260620 06:08:27.998369 22935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:08:27.984665 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50768 (local address 127.11.102.62:37325)
0620 06:08:27.985125 (+ 460us) server_negotiation.cc:206] Beginning negotiation
0620 06:08:27.985144 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0620 06:08:27.985180 (+ 36us) server_negotiation.cc:407] Connection header received
0620 06:08:27.985855 (+ 675us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:08:27.985874 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:08:27.986267 (+ 393us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:08:27.986664 (+ 397us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:08:27.987497 (+ 833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:08:27.988579 (+ 1082us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:08:27.989724 (+ 1145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:08:27.990189 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:08:27.990303 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:08:27.993080 (+ 2777us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:08:27.993087 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:08:27.993105 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:08:27.993141 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:08:27.995070 (+ 1929us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:08:27.995541 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:08:27.995546 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:08:27.995551 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:08:27.995734 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:08:27.996031 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:08:27.996035 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:08:27.996037 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:08:27.996609 (+ 572us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:08:27.996806 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0620 06:08:27.997063 (+ 257us) server_negotiation.cc:299] Negotiation successful
0620 06:08:27.997352 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":307,"thread_start_us":141,"threads_started":1}
I20260620 06:08:27.998355 22938 sys_catalog.cc:458] T 00000000000000000000000000000000 P d7bfab7a8cf74c539a5ada4fe744c741 [sys.catalog]: This master's current role is: LEADER
I20260620 06:08:27.999011 22939 sys_catalog.cc:455] T 00000000000000000000000000000000 P d7bfab7a8cf74c539a5ada4fe744c741 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d7bfab7a8cf74c539a5ada4fe744c741" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d7bfab7a8cf74c539a5ada4fe744c741" member_type: VOTER last_known_addr { host: "127.11.102.62" port: 37325 } } }
I20260620 06:08:27.999111 22939 sys_catalog.cc:458] T 00000000000000000000000000000000 P d7bfab7a8cf74c539a5ada4fe744c741 [sys.catalog]: This master's current role is: LEADER
I20260620 06:08:27.999397 22946 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 06:08:27.999688 22932 ranger_client.cc:318] Using new properties file: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781935603120653-11672-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260620 06:08:28.009284 22946 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 06:08:28.016775 22946 catalog_manager.cc:1347] Generated new cluster ID: a9e94160d56e4164a284edbbf267c0c7
I20260620 06:08:28.016849 22946 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 06:08:28.041477 22946 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 06:08:28.042869 22946 catalog_manager.cc:1504] Loading token signing keys...
I20260620 06:08:28.050460 22946 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d7bfab7a8cf74c539a5ada4fe744c741: Generated new TSK 0
I20260620 06:08:28.051277 22946 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.11.102.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.102.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.102.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781935603120653-11672-0/krb5kdc/kudu_127.11.102.1.keytab.
Entry for principal kudu/127.11.102.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781935603120653-11672-0/krb5kdc/kudu_127.11.102.1.keytab.
I20260620 06:08:28.715022 11672 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9HMp43/build/debug/bin/kudu
/tmp/dist-test-task9HMp43/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781935603120653-11672-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781935603120653-11672-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781935603120653-11672-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781935603120653-11672-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.11.102.20:40509/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.102.1:0
--local_ip_for_outbound_sockets=127.11.102.1
--webserver_interface=127.11.102.1
--webserver_port=0
--tserver_master_addrs=127.11.102.62:37325
--keytab_file=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781935603120653-11672-0/krb5kdc/kudu_127.11.102.1.keytab
--principal=kudu/127.11.102.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.102.20:35851
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781935603120653-11672-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781935603120653-11672-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781935603120653-11672-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 06:08:28.857275 22977 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:08:28.857632 22977 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:08:28.857695 22977 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:08:28.863312 22977 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:08:28.863466 22977 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:08:28.863605 22977 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.102.1
I20260620 06:08:28.870843 22977 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.102.20:35851
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781935603120653-11672-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781935603120653-11672-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.11.102.20:40509/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781935603120653-11672-0/krb5kdc/kudu_127.11.102.1.keytab
--principal=kudu/127.11.102.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.102.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781935603120653-11672-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.11.102.1
--webserver_port=0
--tserver_master_addrs=127.11.102.62:37325
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22977
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.102.1
--log_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781935603120653-11672-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 05:28:48 UTC on ccycloud.quasar-fixhms.root.comops.site
build id 4457020
I20260620 06:08:28.872107 22977 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:08:28.873214 22977 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 06:08:28.885141 22985 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:08:28.885169 22984 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:08:28.885555 22987 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:08:28.886230 22977 server_base.cc:1034] running on GCE node
I20260620 06:08:28.886720 22977 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:08:28.887533 22977 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:08:28.888738 22977 hybrid_clock.cc:648] HybridClock initialized: now 1781935708888704 us; error 47 us; skew 500 ppm
Jun 20 06:08:28 dist-test-slave-9hm5 krb5kdc[22420](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935708, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.102.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 06:08:28.892830 22977 init.cc:377] Logged in from keytab as kudu/127.11.102.1@KRBTEST.COM (short username kudu)
I20260620 06:08:28.893999 22977 webserver.cc:460] Webserver started at http://127.11.102.1:38899/ using document root <none> and password file <none>
I20260620 06:08:28.894739 22977 fs_manager.cc:359] Metadata directory not provided
I20260620 06:08:28.894836 22977 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:08:28.895084 22977 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 06:08:28 dist-test-slave-9hm5 krb5kdc[22420](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935708, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.102.1@KRBTEST.COM for HTTP/127.11.102.20@KRBTEST.COM
I20260620 06:08:28.913574 22977 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781935603120653-11672-0/minicluster-data/ts-0/data/instance:
uuid: "481b27dada144085a879c273ed236c08"
format_stamp: "Formatted at 2026-06-20 06:08:28 on dist-test-slave-9hm5"
server_key: "4a9b70b66a394977ad564d45b6deabea"
server_key_iv: "06fc0b85382721e6e8a0f44063e17bfe"
server_key_version: "kuduclusterkey@0"
I20260620 06:08:28.914260 22977 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781935603120653-11672-0/minicluster-data/ts-0/wal/instance:
uuid: "481b27dada144085a879c273ed236c08"
format_stamp: "Formatted at 2026-06-20 06:08:28 on dist-test-slave-9hm5"
server_key: "4a9b70b66a394977ad564d45b6deabea"
server_key_iv: "06fc0b85382721e6e8a0f44063e17bfe"
server_key_version: "kuduclusterkey@0"
I20260620 06:08:28.919077 22977 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260620 06:08:28.935999 22994 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:08:28.937748 22977 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260620 06:08:28.937881 22977 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781935603120653-11672-0/minicluster-data/ts-0/data,/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781935603120653-11672-0/minicluster-data/ts-0/wal
uuid: "481b27dada144085a879c273ed236c08"
format_stamp: "Formatted at 2026-06-20 06:08:28 on dist-test-slave-9hm5"
server_key: "4a9b70b66a394977ad564d45b6deabea"
server_key_iv: "06fc0b85382721e6e8a0f44063e17bfe"
server_key_version: "kuduclusterkey@0"
I20260620 06:08:28.938021 22977 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781935603120653-11672-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781935603120653-11672-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781935603120653-11672-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:08:28.971041 22977 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:08:28.974471 22977 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:08:28.974668 22977 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:08:28.975361 22977 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 06:08:28.976321 22977 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 06:08:28.976395 22977 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 06:08:28.976501 22977 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 06:08:28.976608 22977 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 06:08:28.987812 23109 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.102.1:46267 every 8 connection(s)
I20260620 06:08:28.987887 22977 rpc_server.cc:307] RPC server started. Bound to: 127.11.102.1:46267
I20260620 06:08:28.989097 22977 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781935603120653-11672-0/minicluster-data/ts-0/data/info.pb
I20260620 06:08:28.991907 11672 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9HMp43/build/debug/bin/kudu as pid 22977
I20260620 06:08:28.992036 11672 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781935603120653-11672-0/minicluster-data/ts-0/wal/instance
I20260620 06:08:28.992328 11672 external_mini_cluster.cc:1428] Setting key 60b15a9c4013635d877c676f9cf481c0
WARNING: no policy specified for kudu/127.11.102.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.102.2@KRBTEST.COM" created.
Jun 20 06:08:29 dist-test-slave-9hm5 krb5kdc[22420](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935708, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.102.1@KRBTEST.COM for kudu/127.11.102.62@KRBTEST.COM
I20260620 06:08:29.015329 23112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:08:28.991063 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.102.1:57083 (local address 127.11.102.62:37325)
0620 06:08:28.991338 (+ 275us) server_negotiation.cc:206] Beginning negotiation
0620 06:08:28.991346 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 06:08:29.000131 (+ 8785us) server_negotiation.cc:407] Connection header received
0620 06:08:29.001374 (+ 1243us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:08:29.001377 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:08:29.001435 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:08:29.001499 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:08:29.003167 (+ 1668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:08:29.004045 (+ 878us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:08:29.005722 (+ 1677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:08:29.006098 (+ 376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:08:29.006172 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:08:29.010961 (+ 4789us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:08:29.010964 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:08:29.010967 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:08:29.010997 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:08:29.012988 (+ 1991us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:08:29.013575 (+ 587us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:08:29.013579 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:08:29.013581 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:08:29.013671 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:08:29.014072 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:08:29.014075 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:08:29.014077 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:08:29.014279 (+ 202us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:08:29.014345 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0620 06:08:29.015014 (+ 669us) server_negotiation.cc:299] Negotiation successful
0620 06:08:29.015141 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":200,"thread_start_us":101,"threads_started":1}
I20260620 06:08:29.016948 23113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:08:28.991524 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.102.62:37325 (local address 127.11.102.1:57083)
0620 06:08:28.999964 (+ 8440us) negotiation.cc:107] Waiting for socket to connect
0620 06:08:29.000029 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0620 06:08:29.001124 (+ 1095us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:08:29.001931 (+ 807us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:08:29.001940 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:08:29.002299 (+ 359us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:08:29.002999 (+ 700us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:08:29.003018 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:08:29.004202 (+ 1184us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:08:29.004205 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:08:29.005585 (+ 1380us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:08:29.005595 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:08:29.006374 (+ 779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:08:29.006378 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:08:29.006570 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:08:29.007262 (+ 692us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:08:29.007282 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:08:29.010760 (+ 3478us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:08:29.013112 (+ 2352us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:08:29.013118 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:08:29.013134 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:08:29.013457 (+ 323us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:08:29.013781 (+ 324us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:08:29.013784 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:08:29.013786 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:08:29.013965 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:08:29.014423 (+ 458us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:08:29.014429 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:08:29.014722 (+ 293us) client_negotiation.cc:769] Sending connection context
0620 06:08:29.015565 (+ 843us) client_negotiation.cc:240] Negotiation successful
0620 06:08:29.015935 (+ 370us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8186,"thread_start_us":124,"threads_started":1}
I20260620 06:08:29.018522 23110 heartbeater.cc:347] Connected to a master server at 127.11.102.62:37325
I20260620 06:08:29.018775 23110 heartbeater.cc:464] Registering TS with master...
I20260620 06:08:29.019346 23110 heartbeater.cc:511] Master 127.11.102.62:37325 requested a full tablet report, sending...
I20260620 06:08:29.021224 22896 ts_manager.cc:194] Registered new tserver with Master: 481b27dada144085a879c273ed236c08 (127.11.102.1:46267)
I20260620 06:08:29.022792 22896 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.102.1@KRBTEST.COM'} at 127.11.102.1:57083
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.102.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781935603120653-11672-0/krb5kdc/kudu_127.11.102.2.keytab.
Entry for principal kudu/127.11.102.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781935603120653-11672-0/krb5kdc/kudu_127.11.102.2.keytab.
I20260620 06:08:29.031200 11672 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9HMp43/build/debug/bin/kudu
/tmp/dist-test-task9HMp43/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781935603120653-11672-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781935603120653-11672-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781935603120653-11672-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781935603120653-11672-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.11.102.20:40509/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.102.2:0
--local_ip_for_outbound_sockets=127.11.102.2
--webserver_interface=127.11.102.2
--webserver_port=0
--tserver_master_addrs=127.11.102.62:37325
--keytab_file=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781935603120653-11672-0/krb5kdc/kudu_127.11.102.2.keytab
--principal=kudu/127.11.102.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.102.20:35851
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781935603120653-11672-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781935603120653-11672-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781935603120653-11672-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 06:08:29 dist-test-slave-9hm5 krb5kdc[22420](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781935709, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.102.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260620 06:08:29.194531 23116 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:08:29.194928 23116 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:08:29.195005 23116 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:08:29.201174 23116 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:08:29.201284 23116 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:08:29.201404 23116 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.102.2
I20260620 06:08:29.210067 23116 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.102.20:35851
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781935603120653-11672-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781935603120653-11672-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.11.102.20:40509/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781935603120653-11672-0/krb5kdc/kudu_127.11.102.2.keytab
--principal=kudu/127.11.102.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.102.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781935603120653-11672-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.11.102.2
--webserver_port=0
--tserver_master_addrs=127.11.102.62:37325
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23116
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.102.2
--log_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781935603120653-11672-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 05:28:48 UTC on ccycloud.quasar-fixhms.root.comops.site
build id 4457020
I20260620 06:08:29.211522 23116 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:08:29.212935 23116 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 06:08:29.225152 23123 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:08:29.225665 23122 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:08:29.226365 23125 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:08:29.226610 23116 server_base.cc:1034] running on GCE node
I20260620 06:08:29.227281 23116 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:08:29.228099 23116 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:08:29.231369 23116 hybrid_clock.cc:648] HybridClock initialized: now 1781935709231345 us; error 36 us; skew 500 ppm
Jun 20 06:08:29 dist-test-slave-9hm5 krb5kdc[22420](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935709, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.102.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 06:08:29.234951 23116 init.cc:377] Logged in from keytab as kudu/127.11.102.2@KRBTEST.COM (short username kudu)
I20260620 06:08:29.236045 23116 webserver.cc:460] Webserver started at http://127.11.102.2:33953/ using document root <none> and password file <none>
I20260620 06:08:29.236718 23116 fs_manager.cc:359] Metadata directory not provided
I20260620 06:08:29.236802 23116 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:08:29.237030 23116 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 06:08:29 dist-test-slave-9hm5 krb5kdc[22420](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935709, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.102.2@KRBTEST.COM for HTTP/127.11.102.20@KRBTEST.COM
I20260620 06:08:29.256486 23116 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781935603120653-11672-0/minicluster-data/ts-1/data/instance:
uuid: "9a3cb170e246427c803511b585170a0f"
format_stamp: "Formatted at 2026-06-20 06:08:29 on dist-test-slave-9hm5"
server_key: "d091501ca6749da80f4100021b0e7e9a"
server_key_iv: "3ce79234999c1354da71b0c402c069cb"
server_key_version: "kuduclusterkey@0"
I20260620 06:08:29.257212 23116 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781935603120653-11672-0/minicluster-data/ts-1/wal/instance:
uuid: "9a3cb170e246427c803511b585170a0f"
format_stamp: "Formatted at 2026-06-20 06:08:29 on dist-test-slave-9hm5"
server_key: "d091501ca6749da80f4100021b0e7e9a"
server_key_iv: "3ce79234999c1354da71b0c402c069cb"
server_key_version: "kuduclusterkey@0"
I20260620 06:08:29.262228 23116 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 06:08:29.289474 23132 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:08:29.291059 23116 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260620 06:08:29.291180 23116 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781935603120653-11672-0/minicluster-data/ts-1/data,/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781935603120653-11672-0/minicluster-data/ts-1/wal
uuid: "9a3cb170e246427c803511b585170a0f"
format_stamp: "Formatted at 2026-06-20 06:08:29 on dist-test-slave-9hm5"
server_key: "d091501ca6749da80f4100021b0e7e9a"
server_key_iv: "3ce79234999c1354da71b0c402c069cb"
server_key_version: "kuduclusterkey@0"
I20260620 06:08:29.291314 23116 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781935603120653-11672-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781935603120653-11672-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781935603120653-11672-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:08:29.311113 23116 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:08:29.314574 23116 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:08:29.314762 23116 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:08:29.315375 23116 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 06:08:29.316368 23116 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 06:08:29.316430 23116 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 06:08:29.316484 23116 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 06:08:29.316514 23116 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 06:08:29.329010 23116 rpc_server.cc:307] RPC server started. Bound to: 127.11.102.2:38369
I20260620 06:08:29.329169 23245 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.102.2:38369 every 8 connection(s)
I20260620 06:08:29.330173 23116 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781935603120653-11672-0/minicluster-data/ts-1/data/info.pb
I20260620 06:08:29.340153 11672 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9HMp43/build/debug/bin/kudu as pid 23116
I20260620 06:08:29.340293 11672 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781935603120653-11672-0/minicluster-data/ts-1/wal/instance
I20260620 06:08:29.340610 11672 external_mini_cluster.cc:1428] Setting key fabb7a368c5eb782256b2a28312454b0
Jun 20 06:08:29 dist-test-slave-9hm5 krb5kdc[22420](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935709, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.102.2@KRBTEST.COM for kudu/127.11.102.62@KRBTEST.COM
I20260620 06:08:29.349335 23112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:08:29.333898 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.102.2:34565 (local address 127.11.102.62:37325)
0620 06:08:29.334028 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0620 06:08:29.334038 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 06:08:29.334661 (+ 623us) server_negotiation.cc:407] Connection header received
0620 06:08:29.336014 (+ 1353us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:08:29.336019 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:08:29.336078 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:08:29.336137 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:08:29.337810 (+ 1673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:08:29.338655 (+ 845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:08:29.341438 (+ 2783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:08:29.341766 (+ 328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:08:29.341874 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:08:29.344790 (+ 2916us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:08:29.344795 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:08:29.344799 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:08:29.344829 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:08:29.346927 (+ 2098us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:08:29.347516 (+ 589us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:08:29.347520 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:08:29.347521 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:08:29.347636 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:08:29.348040 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:08:29.348043 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:08:29.348045 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:08:29.348259 (+ 214us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:08:29.348340 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0620 06:08:29.349028 (+ 688us) server_negotiation.cc:299] Negotiation successful
0620 06:08:29.349166 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
WARNING: no policy specified for kudu/127.11.102.3@KRBTEST.COM; defaulting to no policy
I20260620 06:08:29.350939 23248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:08:29.332759 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.102.62:37325 (local address 127.11.102.2:34565)
0620 06:08:29.334466 (+ 1707us) negotiation.cc:107] Waiting for socket to connect
0620 06:08:29.334525 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0620 06:08:29.335742 (+ 1217us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:08:29.336572 (+ 830us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:08:29.336582 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:08:29.336959 (+ 377us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:08:29.337618 (+ 659us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:08:29.337639 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:08:29.338807 (+ 1168us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:08:29.338812 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:08:29.341304 (+ 2492us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:08:29.341314 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:08:29.342043 (+ 729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:08:29.342047 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:08:29.342243 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:08:29.342967 (+ 724us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:08:29.342988 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:08:29.344604 (+ 1616us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:08:29.347060 (+ 2456us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:08:29.347066 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:08:29.347082 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:08:29.347403 (+ 321us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:08:29.347759 (+ 356us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:08:29.347761 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:08:29.347763 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:08:29.347931 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:08:29.348406 (+ 475us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:08:29.348412 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:08:29.348731 (+ 319us) client_negotiation.cc:769] Sending connection context
0620 06:08:29.349507 (+ 776us) client_negotiation.cc:240] Negotiation successful
0620 06:08:29.349867 (+ 360us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1478,"thread_start_us":147,"threads_started":1}
I20260620 06:08:29.352341 23246 heartbeater.cc:347] Connected to a master server at 127.11.102.62:37325
I20260620 06:08:29.352666 23246 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.102.3@KRBTEST.COM" created.
I20260620 06:08:29.353296 23246 heartbeater.cc:511] Master 127.11.102.62:37325 requested a full tablet report, sending...
I20260620 06:08:29.354312 22896 ts_manager.cc:194] Registered new tserver with Master: 9a3cb170e246427c803511b585170a0f (127.11.102.2:38369)
I20260620 06:08:29.354885 22896 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.102.2@KRBTEST.COM'} at 127.11.102.2:34565
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.102.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781935603120653-11672-0/krb5kdc/kudu_127.11.102.3.keytab.
Entry for principal kudu/127.11.102.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781935603120653-11672-0/krb5kdc/kudu_127.11.102.3.keytab.
I20260620 06:08:29.382282 11672 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9HMp43/build/debug/bin/kudu
/tmp/dist-test-task9HMp43/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781935603120653-11672-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781935603120653-11672-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781935603120653-11672-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781935603120653-11672-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.11.102.20:40509/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.102.3:0
--local_ip_for_outbound_sockets=127.11.102.3
--webserver_interface=127.11.102.3
--webserver_port=0
--tserver_master_addrs=127.11.102.62:37325
--keytab_file=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781935603120653-11672-0/krb5kdc/kudu_127.11.102.3.keytab
--principal=kudu/127.11.102.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.102.20:35851
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781935603120653-11672-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781935603120653-11672-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781935603120653-11672-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 06:08:29 dist-test-slave-9hm5 krb5kdc[22420](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935709, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.102.62@KRBTEST.COM for HTTP/127.11.102.20@KRBTEST.COM
W20260620 06:08:29.544697 23251 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:08:29.545013 23251 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:08:29.545096 23251 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:08:29.549986 23251 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:08:29.550076 23251 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:08:29.550182 23251 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.102.3
I20260620 06:08:29.555548 23251 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.102.20:35851
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781935603120653-11672-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781935603120653-11672-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.11.102.20:40509/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781935603120653-11672-0/krb5kdc/kudu_127.11.102.3.keytab
--principal=kudu/127.11.102.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.102.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781935603120653-11672-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.11.102.3
--webserver_port=0
--tserver_master_addrs=127.11.102.62:37325
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23251
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.102.3
--log_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781935603120653-11672-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 05:28:48 UTC on ccycloud.quasar-fixhms.root.comops.site
build id 4457020
I20260620 06:08:29.556705 23251 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:08:29.557751 23251 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 06:08:29.568344 23257 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:08:29.568718 23259 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:08:29.568938 23251 server_base.cc:1034] running on GCE node
W20260620 06:08:29.569290 23256 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:08:29.569778 23251 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:08:29.570588 23251 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:08:29.572041 23251 hybrid_clock.cc:648] HybridClock initialized: now 1781935709571969 us; error 88 us; skew 500 ppm
Jun 20 06:08:29 dist-test-slave-9hm5 krb5kdc[22420](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935709, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.102.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 06:08:29.575919 23251 init.cc:377] Logged in from keytab as kudu/127.11.102.3@KRBTEST.COM (short username kudu)
I20260620 06:08:29.577045 23251 webserver.cc:460] Webserver started at http://127.11.102.3:41425/ using document root <none> and password file <none>
I20260620 06:08:29.577785 23251 fs_manager.cc:359] Metadata directory not provided
I20260620 06:08:29.577898 23251 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:08:29.578138 23251 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 06:08:29 dist-test-slave-9hm5 krb5kdc[22420](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935709, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.102.3@KRBTEST.COM for HTTP/127.11.102.20@KRBTEST.COM
I20260620 06:08:29.595726 23251 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781935603120653-11672-0/minicluster-data/ts-2/data/instance:
uuid: "cbe512eba99e411d8948edb8a6d628ee"
format_stamp: "Formatted at 2026-06-20 06:08:29 on dist-test-slave-9hm5"
server_key: "2ec70a462e66f419aea84f63f0225bb1"
server_key_iv: "e276d2cb0e8c1b9fb387541765bb85fb"
server_key_version: "kuduclusterkey@0"
I20260620 06:08:29.596477 23251 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781935603120653-11672-0/minicluster-data/ts-2/wal/instance:
uuid: "cbe512eba99e411d8948edb8a6d628ee"
format_stamp: "Formatted at 2026-06-20 06:08:29 on dist-test-slave-9hm5"
server_key: "2ec70a462e66f419aea84f63f0225bb1"
server_key_iv: "e276d2cb0e8c1b9fb387541765bb85fb"
server_key_version: "kuduclusterkey@0"
I20260620 06:08:29.601802 23251 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.003s
I20260620 06:08:29.640550 23266 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:08:29.651109 23251 fs_manager.cc:725] Time spent opening block manager: real 0.013s user 0.003s sys 0.001s
I20260620 06:08:29.651278 23251 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781935603120653-11672-0/minicluster-data/ts-2/data,/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781935603120653-11672-0/minicluster-data/ts-2/wal
uuid: "cbe512eba99e411d8948edb8a6d628ee"
format_stamp: "Formatted at 2026-06-20 06:08:29 on dist-test-slave-9hm5"
server_key: "2ec70a462e66f419aea84f63f0225bb1"
server_key_iv: "e276d2cb0e8c1b9fb387541765bb85fb"
server_key_version: "kuduclusterkey@0"
I20260620 06:08:29.651440 23251 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781935603120653-11672-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781935603120653-11672-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781935603120653-11672-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:08:29.671459 23251 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:08:29.675325 23251 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:08:29.675541 23251 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:08:29.676394 23251 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 06:08:29.677485 23251 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 06:08:29.677560 23251 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 06:08:29.677627 23251 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 06:08:29.677671 23251 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 06:08:29.692916 23251 rpc_server.cc:307] RPC server started. Bound to: 127.11.102.3:41945
I20260620 06:08:29.694115 23251 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781935603120653-11672-0/minicluster-data/ts-2/data/info.pb
I20260620 06:08:29.702960 11672 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9HMp43/build/debug/bin/kudu as pid 23251
I20260620 06:08:29.703107 11672 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781935603120653-11672-0/minicluster-data/ts-2/wal/instance
I20260620 06:08:29.703473 11672 external_mini_cluster.cc:1428] Setting key 04ed206c044cde3384826549da08719b
I20260620 06:08:29.698136 23379 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.102.3:41945 every 8 connection(s)
Jun 20 06:08:29 dist-test-slave-9hm5 krb5kdc[22420](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935709, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.102.3@KRBTEST.COM for kudu/127.11.102.62@KRBTEST.COM
I20260620 06:08:29.721447 23112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:08:29.704727 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.102.3:37571 (local address 127.11.102.62:37325)
0620 06:08:29.704861 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0620 06:08:29.704866 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 06:08:29.706337 (+ 1471us) server_negotiation.cc:407] Connection header received
0620 06:08:29.707594 (+ 1257us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:08:29.707597 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:08:29.707683 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:08:29.707745 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:08:29.709402 (+ 1657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:08:29.710199 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:08:29.711987 (+ 1788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:08:29.712341 (+ 354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:08:29.712429 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:08:29.715797 (+ 3368us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:08:29.715802 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:08:29.715805 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:08:29.715832 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:08:29.718527 (+ 2695us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:08:29.719421 (+ 894us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:08:29.719426 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:08:29.719428 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:08:29.719522 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:08:29.720035 (+ 513us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:08:29.720039 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:08:29.720041 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:08:29.720284 (+ 243us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:08:29.720386 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0620 06:08:29.721130 (+ 744us) server_negotiation.cc:299] Negotiation successful
0620 06:08:29.721261 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260620 06:08:29.723402 23382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:08:29.701160 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.102.62:37325 (local address 127.11.102.3:37571)
0620 06:08:29.706175 (+ 5015us) negotiation.cc:107] Waiting for socket to connect
0620 06:08:29.706237 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0620 06:08:29.707336 (+ 1099us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:08:29.708180 (+ 844us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:08:29.708191 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:08:29.708547 (+ 356us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:08:29.709229 (+ 682us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:08:29.709252 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:08:29.710366 (+ 1114us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:08:29.710370 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:08:29.711828 (+ 1458us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:08:29.711843 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:08:29.712634 (+ 791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:08:29.712638 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:08:29.712888 (+ 250us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:08:29.713716 (+ 828us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:08:29.713739 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:08:29.715560 (+ 1821us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:08:29.718897 (+ 3337us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:08:29.718904 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:08:29.718920 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:08:29.719271 (+ 351us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:08:29.719706 (+ 435us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:08:29.719709 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:08:29.719711 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:08:29.719906 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:08:29.720463 (+ 557us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:08:29.720470 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:08:29.720816 (+ 346us) client_negotiation.cc:769] Sending connection context
0620 06:08:29.721947 (+ 1131us) client_negotiation.cc:240] Negotiation successful
0620 06:08:29.722297 (+ 350us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4574,"thread_start_us":185,"threads_started":1}
I20260620 06:08:29.725016 23380 heartbeater.cc:347] Connected to a master server at 127.11.102.62:37325
I20260620 06:08:29.725252 23380 heartbeater.cc:464] Registering TS with master...
I20260620 06:08:29.725828 23380 heartbeater.cc:511] Master 127.11.102.62:37325 requested a full tablet report, sending...
I20260620 06:08:29.726998 22896 ts_manager.cc:194] Registered new tserver with Master: cbe512eba99e411d8948edb8a6d628ee (127.11.102.3:41945)
I20260620 06:08:29.727802 22896 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.102.3@KRBTEST.COM'} at 127.11.102.3:37571
I20260620 06:08:29.731719 11672 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 06:08:29.745759 23112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:08:29.734398 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35380 (local address 127.11.102.62:37325)
0620 06:08:29.734527 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0620 06:08:29.734533 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 06:08:29.735239 (+ 706us) server_negotiation.cc:407] Connection header received
0620 06:08:29.735450 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:08:29.735454 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:08:29.735507 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:08:29.735565 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:08:29.736646 (+ 1081us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:08:29.737441 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:08:29.738728 (+ 1287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:08:29.739072 (+ 344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:08:29.739158 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:08:29.741168 (+ 2010us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:08:29.741173 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:08:29.741176 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:08:29.741206 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:08:29.743489 (+ 2283us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:08:29.744101 (+ 612us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:08:29.744105 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:08:29.744108 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:08:29.744203 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:08:29.744569 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:08:29.744573 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:08:29.744575 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:08:29.744871 (+ 296us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:08:29.745036 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0620 06:08:29.745436 (+ 400us) server_negotiation.cc:299] Negotiation successful
0620 06:08:29.745563 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260620 06:08:29.750066 22896 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35380:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 06:08:29.754859 22896 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 06:08:29.788321 23395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:08:29.770200 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.102.3:41945 (local address 127.0.0.1:33578)
0620 06:08:29.774814 (+ 4614us) negotiation.cc:107] Waiting for socket to connect
0620 06:08:29.774830 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 06:08:29.774970 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:08:29.776017 (+ 1047us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:08:29.776024 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:08:29.776047 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 06:08:29.776206 (+ 159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:08:29.776215 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:08:29.781712 (+ 5497us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:08:29.781716 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:08:29.783158 (+ 1442us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:08:29.783170 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:08:29.787724 (+ 4554us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:08:29.787729 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:08:29.787849 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:08:29.787896 (+ 47us) client_negotiation.cc:769] Sending connection context
0620 06:08:29.788012 (+ 116us) client_negotiation.cc:240] Negotiation successful
0620 06:08:29.788144 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4541,"thread_start_us":88,"threads_started":1}
I20260620 06:08:29.789197 23393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:08:29.767898 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51654 (local address 127.11.102.2:38369)
0620 06:08:29.775833 (+ 7935us) server_negotiation.cc:206] Beginning negotiation
0620 06:08:29.775844 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 06:08:29.775868 (+ 24us) server_negotiation.cc:407] Connection header received
0620 06:08:29.783750 (+ 7882us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:08:29.783757 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:08:29.783962 (+ 205us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:08:29.784122 (+ 160us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 06:08:29.784539 (+ 417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:08:29.785706 (+ 1167us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:08:29.787229 (+ 1523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:08:29.788114 (+ 885us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:08:29.788232 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:08:29.788367 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0620 06:08:29.788954 (+ 587us) server_negotiation.cc:299] Negotiation successful
0620 06:08:29.789052 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7825,"thread_start_us":146,"threads_started":1}
I20260620 06:08:29.790357 23398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:08:29.772684 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38204 (local address 127.11.102.1:46267)
0620 06:08:29.774658 (+ 1974us) server_negotiation.cc:206] Beginning negotiation
0620 06:08:29.774666 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 06:08:29.776663 (+ 1997us) server_negotiation.cc:407] Connection header received
0620 06:08:29.776753 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:08:29.776759 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:08:29.776956 (+ 197us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:08:29.777939 (+ 983us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 06:08:29.779792 (+ 1853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:08:29.783428 (+ 3636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:08:29.784961 (+ 1533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:08:29.789191 (+ 4230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:08:29.789313 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:08:29.789454 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0620 06:08:29.790088 (+ 634us) server_negotiation.cc:299] Negotiation successful
0620 06:08:29.790197 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1847,"thread_start_us":87,"threads_started":1}
I20260620 06:08:29.788846 23397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:08:29.768177 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.102.2:38369 (local address 127.0.0.1:51654)
0620 06:08:29.773562 (+ 5385us) negotiation.cc:107] Waiting for socket to connect
0620 06:08:29.773614 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0620 06:08:29.775868 (+ 2254us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:08:29.784268 (+ 8400us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:08:29.784272 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:08:29.784289 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 06:08:29.784406 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:08:29.784412 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:08:29.785842 (+ 1430us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:08:29.785846 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:08:29.787075 (+ 1229us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:08:29.787085 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:08:29.788437 (+ 1352us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:08:29.788441 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:08:29.788542 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:08:29.788564 (+ 22us) client_negotiation.cc:769] Sending connection context
0620 06:08:29.788632 (+ 68us) client_negotiation.cc:240] Negotiation successful
0620 06:08:29.788704 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5211,"thread_start_us":105,"threads_started":1}
I20260620 06:08:29.789968 23394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:08:29.772218 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.102.1:46267 (local address 127.0.0.1:38204)
0620 06:08:29.776434 (+ 4216us) negotiation.cc:107] Waiting for socket to connect
0620 06:08:29.776449 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 06:08:29.776544 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:08:29.779475 (+ 2931us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:08:29.779478 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:08:29.779499 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 06:08:29.779634 (+ 135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:08:29.779641 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:08:29.783573 (+ 3932us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:08:29.783577 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:08:29.784797 (+ 1220us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:08:29.784810 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:08:29.789530 (+ 4720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:08:29.789535 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:08:29.789640 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:08:29.789681 (+ 41us) client_negotiation.cc:769] Sending connection context
0620 06:08:29.789754 (+ 73us) client_negotiation.cc:240] Negotiation successful
0620 06:08:29.789824 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4155,"thread_start_us":94,"threads_started":1}
I20260620 06:08:29.792568 23396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:08:29.770606 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33578 (local address 127.11.102.3:41945)
0620 06:08:29.773301 (+ 2695us) server_negotiation.cc:206] Beginning negotiation
0620 06:08:29.773317 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0620 06:08:29.775117 (+ 1800us) server_negotiation.cc:407] Connection header received
0620 06:08:29.775209 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:08:29.775214 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:08:29.775447 (+ 233us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:08:29.775673 (+ 226us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 06:08:29.778113 (+ 2440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:08:29.779319 (+ 1206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:08:29.786412 (+ 7093us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:08:29.787312 (+ 900us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:08:29.787446 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:08:29.787652 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0620 06:08:29.790938 (+ 3286us) server_negotiation.cc:299] Negotiation successful
0620 06:08:29.791050 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2579,"thread_start_us":128,"threads_started":1}
I20260620 06:08:29.793951 23175 tablet_service.cc:1467] Processing CreateTablet for tablet 9be73ad9ceec422492ce89080293be29 (DEFAULT_TABLE table=test-table [id=8a8143bc6851402eb8e7b957aeafbccd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 06:08:29.794375 23026 tablet_service.cc:1467] Processing CreateTablet for tablet 9be73ad9ceec422492ce89080293be29 (DEFAULT_TABLE table=test-table [id=8a8143bc6851402eb8e7b957aeafbccd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 06:08:29.795105 23175 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9be73ad9ceec422492ce89080293be29. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 06:08:29.795359 23026 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9be73ad9ceec422492ce89080293be29. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 06:08:29.795231 23313 tablet_service.cc:1467] Processing CreateTablet for tablet 9be73ad9ceec422492ce89080293be29 (DEFAULT_TABLE table=test-table [id=8a8143bc6851402eb8e7b957aeafbccd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 06:08:29.796157 23313 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9be73ad9ceec422492ce89080293be29. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 06:08:29.804533 23401 tablet_bootstrap.cc:492] T 9be73ad9ceec422492ce89080293be29 P cbe512eba99e411d8948edb8a6d628ee: Bootstrap starting.
I20260620 06:08:29.807404 23401 tablet_bootstrap.cc:654] T 9be73ad9ceec422492ce89080293be29 P cbe512eba99e411d8948edb8a6d628ee: Neither blocks nor log segments found. Creating new log.
I20260620 06:08:29.808315 23401 log.cc:824] T 9be73ad9ceec422492ce89080293be29 P cbe512eba99e411d8948edb8a6d628ee: Log is configured to *not* fsync() on all Append() calls
I20260620 06:08:29.809476 23400 tablet_bootstrap.cc:492] T 9be73ad9ceec422492ce89080293be29 P 9a3cb170e246427c803511b585170a0f: Bootstrap starting.
I20260620 06:08:29.810500 23401 tablet_bootstrap.cc:492] T 9be73ad9ceec422492ce89080293be29 P cbe512eba99e411d8948edb8a6d628ee: No bootstrap required, opened a new log
I20260620 06:08:29.810678 23401 ts_tablet_manager.cc:1397] T 9be73ad9ceec422492ce89080293be29 P cbe512eba99e411d8948edb8a6d628ee: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.005s
I20260620 06:08:29.812237 23400 tablet_bootstrap.cc:654] T 9be73ad9ceec422492ce89080293be29 P 9a3cb170e246427c803511b585170a0f: Neither blocks nor log segments found. Creating new log.
I20260620 06:08:29.813577 23400 log.cc:824] T 9be73ad9ceec422492ce89080293be29 P 9a3cb170e246427c803511b585170a0f: Log is configured to *not* fsync() on all Append() calls
I20260620 06:08:29.814342 23399 tablet_bootstrap.cc:492] T 9be73ad9ceec422492ce89080293be29 P 481b27dada144085a879c273ed236c08: Bootstrap starting.
I20260620 06:08:29.814740 23401 raft_consensus.cc:348] T 9be73ad9ceec422492ce89080293be29 P cbe512eba99e411d8948edb8a6d628ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3cb170e246427c803511b585170a0f" member_type: VOTER last_known_addr { host: "127.11.102.2" port: 38369 } } peers { permanent_uuid: "cbe512eba99e411d8948edb8a6d628ee" member_type: VOTER last_known_addr { host: "127.11.102.3" port: 41945 } } peers { permanent_uuid: "481b27dada144085a879c273ed236c08" member_type: VOTER last_known_addr { host: "127.11.102.1" port: 46267 } }
I20260620 06:08:29.814997 23401 raft_consensus.cc:374] T 9be73ad9ceec422492ce89080293be29 P cbe512eba99e411d8948edb8a6d628ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 06:08:29.815050 23401 raft_consensus.cc:729] T 9be73ad9ceec422492ce89080293be29 P cbe512eba99e411d8948edb8a6d628ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cbe512eba99e411d8948edb8a6d628ee, State: Initialized, Role: FOLLOWER
I20260620 06:08:29.815526 23401 consensus_queue.cc:260] T 9be73ad9ceec422492ce89080293be29 P cbe512eba99e411d8948edb8a6d628ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3cb170e246427c803511b585170a0f" member_type: VOTER last_known_addr { host: "127.11.102.2" port: 38369 } } peers { permanent_uuid: "cbe512eba99e411d8948edb8a6d628ee" member_type: VOTER last_known_addr { host: "127.11.102.3" port: 41945 } } peers { permanent_uuid: "481b27dada144085a879c273ed236c08" member_type: VOTER last_known_addr { host: "127.11.102.1" port: 46267 } }
I20260620 06:08:29.816721 23399 tablet_bootstrap.cc:654] T 9be73ad9ceec422492ce89080293be29 P 481b27dada144085a879c273ed236c08: Neither blocks nor log segments found. Creating new log.
I20260620 06:08:29.817595 23399 log.cc:824] T 9be73ad9ceec422492ce89080293be29 P 481b27dada144085a879c273ed236c08: Log is configured to *not* fsync() on all Append() calls
I20260620 06:08:29.817778 23401 ts_tablet_manager.cc:1428] T 9be73ad9ceec422492ce89080293be29 P cbe512eba99e411d8948edb8a6d628ee: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260620 06:08:29.818159 23380 heartbeater.cc:503] Master 127.11.102.62:37325 was elected leader, sending a full tablet report...
I20260620 06:08:29.819900 23400 tablet_bootstrap.cc:492] T 9be73ad9ceec422492ce89080293be29 P 9a3cb170e246427c803511b585170a0f: No bootstrap required, opened a new log
I20260620 06:08:29.820163 23400 ts_tablet_manager.cc:1397] T 9be73ad9ceec422492ce89080293be29 P 9a3cb170e246427c803511b585170a0f: Time spent bootstrapping tablet: real 0.011s user 0.002s sys 0.005s
I20260620 06:08:29.825549 23399 tablet_bootstrap.cc:492] T 9be73ad9ceec422492ce89080293be29 P 481b27dada144085a879c273ed236c08: No bootstrap required, opened a new log
I20260620 06:08:29.825842 23399 ts_tablet_manager.cc:1397] T 9be73ad9ceec422492ce89080293be29 P 481b27dada144085a879c273ed236c08: Time spent bootstrapping tablet: real 0.012s user 0.005s sys 0.000s
I20260620 06:08:29.826928 23400 raft_consensus.cc:348] T 9be73ad9ceec422492ce89080293be29 P 9a3cb170e246427c803511b585170a0f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3cb170e246427c803511b585170a0f" member_type: VOTER last_known_addr { host: "127.11.102.2" port: 38369 } } peers { permanent_uuid: "cbe512eba99e411d8948edb8a6d628ee" member_type: VOTER last_known_addr { host: "127.11.102.3" port: 41945 } } peers { permanent_uuid: "481b27dada144085a879c273ed236c08" member_type: VOTER last_known_addr { host: "127.11.102.1" port: 46267 } }
I20260620 06:08:29.827303 23400 raft_consensus.cc:374] T 9be73ad9ceec422492ce89080293be29 P 9a3cb170e246427c803511b585170a0f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 06:08:29.827397 23400 raft_consensus.cc:729] T 9be73ad9ceec422492ce89080293be29 P 9a3cb170e246427c803511b585170a0f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a3cb170e246427c803511b585170a0f, State: Initialized, Role: FOLLOWER
I20260620 06:08:29.827951 23400 consensus_queue.cc:260] T 9be73ad9ceec422492ce89080293be29 P 9a3cb170e246427c803511b585170a0f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3cb170e246427c803511b585170a0f" member_type: VOTER last_known_addr { host: "127.11.102.2" port: 38369 } } peers { permanent_uuid: "cbe512eba99e411d8948edb8a6d628ee" member_type: VOTER last_known_addr { host: "127.11.102.3" port: 41945 } } peers { permanent_uuid: "481b27dada144085a879c273ed236c08" member_type: VOTER last_known_addr { host: "127.11.102.1" port: 46267 } }
I20260620 06:08:29.828680 23246 heartbeater.cc:503] Master 127.11.102.62:37325 was elected leader, sending a full tablet report...
I20260620 06:08:29.829079 23400 ts_tablet_manager.cc:1428] T 9be73ad9ceec422492ce89080293be29 P 9a3cb170e246427c803511b585170a0f: Time spent starting tablet: real 0.009s user 0.006s sys 0.000s
I20260620 06:08:29.833110 23399 raft_consensus.cc:348] T 9be73ad9ceec422492ce89080293be29 P 481b27dada144085a879c273ed236c08 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3cb170e246427c803511b585170a0f" member_type: VOTER last_known_addr { host: "127.11.102.2" port: 38369 } } peers { permanent_uuid: "cbe512eba99e411d8948edb8a6d628ee" member_type: VOTER last_known_addr { host: "127.11.102.3" port: 41945 } } peers { permanent_uuid: "481b27dada144085a879c273ed236c08" member_type: VOTER last_known_addr { host: "127.11.102.1" port: 46267 } }
I20260620 06:08:29.833611 23399 raft_consensus.cc:374] T 9be73ad9ceec422492ce89080293be29 P 481b27dada144085a879c273ed236c08 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 06:08:29.833735 23399 raft_consensus.cc:729] T 9be73ad9ceec422492ce89080293be29 P 481b27dada144085a879c273ed236c08 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 481b27dada144085a879c273ed236c08, State: Initialized, Role: FOLLOWER
W20260620 06:08:29.834050 23247 tablet.cc:2369] T 9be73ad9ceec422492ce89080293be29 P 9a3cb170e246427c803511b585170a0f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 06:08:29.834295 23399 consensus_queue.cc:260] T 9be73ad9ceec422492ce89080293be29 P 481b27dada144085a879c273ed236c08 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3cb170e246427c803511b585170a0f" member_type: VOTER last_known_addr { host: "127.11.102.2" port: 38369 } } peers { permanent_uuid: "cbe512eba99e411d8948edb8a6d628ee" member_type: VOTER last_known_addr { host: "127.11.102.3" port: 41945 } } peers { permanent_uuid: "481b27dada144085a879c273ed236c08" member_type: VOTER last_known_addr { host: "127.11.102.1" port: 46267 } }
I20260620 06:08:29.835438 23399 ts_tablet_manager.cc:1428] T 9be73ad9ceec422492ce89080293be29 P 481b27dada144085a879c273ed236c08: Time spent starting tablet: real 0.009s user 0.006s sys 0.000s
I20260620 06:08:29.835844 23110 heartbeater.cc:503] Master 127.11.102.62:37325 was elected leader, sending a full tablet report...
Jun 20 06:08:29 dist-test-slave-9hm5 krb5kdc[22420](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935709, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.102.62@KRBTEST.COM for HTTP/127.11.102.20@KRBTEST.COM
I20260620 06:08:29.942451 23406 raft_consensus.cc:482] T 9be73ad9ceec422492ce89080293be29 P 9a3cb170e246427c803511b585170a0f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 06:08:29.942699 23406 raft_consensus.cc:504] T 9be73ad9ceec422492ce89080293be29 P 9a3cb170e246427c803511b585170a0f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3cb170e246427c803511b585170a0f" member_type: VOTER last_known_addr { host: "127.11.102.2" port: 38369 } } peers { permanent_uuid: "cbe512eba99e411d8948edb8a6d628ee" member_type: VOTER last_known_addr { host: "127.11.102.3" port: 41945 } } peers { permanent_uuid: "481b27dada144085a879c273ed236c08" member_type: VOTER last_known_addr { host: "127.11.102.1" port: 46267 } }
I20260620 06:08:29.944106 23406 leader_election.cc:288] T 9be73ad9ceec422492ce89080293be29 P 9a3cb170e246427c803511b585170a0f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cbe512eba99e411d8948edb8a6d628ee (127.11.102.3:41945), 481b27dada144085a879c273ed236c08 (127.11.102.1:46267)
I20260620 06:08:29.947978 23407 raft_consensus.cc:482] T 9be73ad9ceec422492ce89080293be29 P 481b27dada144085a879c273ed236c08 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 06:08:29.948202 23407 raft_consensus.cc:504] T 9be73ad9ceec422492ce89080293be29 P 481b27dada144085a879c273ed236c08 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3cb170e246427c803511b585170a0f" member_type: VOTER last_known_addr { host: "127.11.102.2" port: 38369 } } peers { permanent_uuid: "cbe512eba99e411d8948edb8a6d628ee" member_type: VOTER last_known_addr { host: "127.11.102.3" port: 41945 } } peers { permanent_uuid: "481b27dada144085a879c273ed236c08" member_type: VOTER last_known_addr { host: "127.11.102.1" port: 46267 } }
I20260620 06:08:29.949576 23407 leader_election.cc:288] T 9be73ad9ceec422492ce89080293be29 P 481b27dada144085a879c273ed236c08 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9a3cb170e246427c803511b585170a0f (127.11.102.2:38369), cbe512eba99e411d8948edb8a6d628ee (127.11.102.3:41945)
I20260620 06:08:29.955597 23398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:08:29.944924 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.102.2:48795 (local address 127.11.102.1:46267)
0620 06:08:29.945049 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0620 06:08:29.945054 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 06:08:29.951288 (+ 6234us) server_negotiation.cc:407] Connection header received
0620 06:08:29.951462 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:08:29.951465 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:08:29.951509 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:08:29.951555 (+ 46us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 06:08:29.951982 (+ 427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:08:29.952758 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:08:29.954324 (+ 1566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:08:29.954935 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:08:29.955030 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:08:29.955082 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0620 06:08:29.955405 (+ 323us) server_negotiation.cc:299] Negotiation successful
0620 06:08:29.955480 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
W20260620 06:08:29.956003 23381 tablet.cc:2369] T 9be73ad9ceec422492ce89080293be29 P cbe512eba99e411d8948edb8a6d628ee: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 06:08:29.956980 23411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:08:29.944481 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.102.1:46267 (local address 127.11.102.2:48795)
0620 06:08:29.951204 (+ 6723us) negotiation.cc:107] Waiting for socket to connect
0620 06:08:29.951217 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 06:08:29.951348 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:08:29.951727 (+ 379us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:08:29.951730 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:08:29.951747 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 06:08:29.951858 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:08:29.951864 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:08:29.952897 (+ 1033us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:08:29.952900 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:08:29.954186 (+ 1286us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:08:29.954197 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:08:29.955182 (+ 985us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:08:29.955185 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:08:29.955270 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:08:29.955293 (+ 23us) client_negotiation.cc:769] Sending connection context
0620 06:08:29.956724 (+ 1431us) client_negotiation.cc:240] Negotiation successful
0620 06:08:29.956812 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6662,"thread_start_us":144,"threads_started":1}
I20260620 06:08:29.957690 23046 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9be73ad9ceec422492ce89080293be29" candidate_uuid: "9a3cb170e246427c803511b585170a0f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "481b27dada144085a879c273ed236c08" is_pre_election: true
I20260620 06:08:29.957933 23046 raft_consensus.cc:2454] T 9be73ad9ceec422492ce89080293be29 P 481b27dada144085a879c273ed236c08 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9a3cb170e246427c803511b585170a0f in term 0.
I20260620 06:08:29.958446 23135 leader_election.cc:302] T 9be73ad9ceec422492ce89080293be29 P 9a3cb170e246427c803511b585170a0f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 481b27dada144085a879c273ed236c08, 9a3cb170e246427c803511b585170a0f; no voters:
I20260620 06:08:29.958806 23406 raft_consensus.cc:2781] T 9be73ad9ceec422492ce89080293be29 P 9a3cb170e246427c803511b585170a0f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 06:08:29.958896 23406 raft_consensus.cc:482] T 9be73ad9ceec422492ce89080293be29 P 9a3cb170e246427c803511b585170a0f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 06:08:29.958940 23406 raft_consensus.cc:3037] T 9be73ad9ceec422492ce89080293be29 P 9a3cb170e246427c803511b585170a0f [term 0 FOLLOWER]: Advancing to term 1
I20260620 06:08:29.960350 23406 raft_consensus.cc:504] T 9be73ad9ceec422492ce89080293be29 P 9a3cb170e246427c803511b585170a0f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3cb170e246427c803511b585170a0f" member_type: VOTER last_known_addr { host: "127.11.102.2" port: 38369 } } peers { permanent_uuid: "cbe512eba99e411d8948edb8a6d628ee" member_type: VOTER last_known_addr { host: "127.11.102.3" port: 41945 } } peers { permanent_uuid: "481b27dada144085a879c273ed236c08" member_type: VOTER last_known_addr { host: "127.11.102.1" port: 46267 } }
I20260620 06:08:29.960732 23406 leader_election.cc:288] T 9be73ad9ceec422492ce89080293be29 P 9a3cb170e246427c803511b585170a0f [CANDIDATE]: Term 1 election: Requested vote from peers cbe512eba99e411d8948edb8a6d628ee (127.11.102.3:41945), 481b27dada144085a879c273ed236c08 (127.11.102.1:46267)
I20260620 06:08:29.961222 23046 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9be73ad9ceec422492ce89080293be29" candidate_uuid: "9a3cb170e246427c803511b585170a0f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "481b27dada144085a879c273ed236c08"
I20260620 06:08:29.961347 23046 raft_consensus.cc:3037] T 9be73ad9ceec422492ce89080293be29 P 481b27dada144085a879c273ed236c08 [term 0 FOLLOWER]: Advancing to term 1
I20260620 06:08:29.962563 23396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:08:29.947769 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.102.2:44159 (local address 127.11.102.3:41945)
0620 06:08:29.947906 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0620 06:08:29.947911 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 06:08:29.950905 (+ 2994us) server_negotiation.cc:407] Connection header received
0620 06:08:29.951056 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:08:29.951059 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:08:29.951111 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:08:29.951190 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 06:08:29.957263 (+ 6073us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:08:29.957947 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:08:29.961140 (+ 3193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:08:29.961770 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:08:29.961867 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:08:29.961919 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0620 06:08:29.962328 (+ 409us) server_negotiation.cc:299] Negotiation successful
0620 06:08:29.962435 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260620 06:08:29.962699 23046 raft_consensus.cc:2454] T 9be73ad9ceec422492ce89080293be29 P 481b27dada144085a879c273ed236c08 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9a3cb170e246427c803511b585170a0f in term 1.
I20260620 06:08:29.963047 23135 leader_election.cc:302] T 9be73ad9ceec422492ce89080293be29 P 9a3cb170e246427c803511b585170a0f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 481b27dada144085a879c273ed236c08, 9a3cb170e246427c803511b585170a0f; no voters:
I20260620 06:08:29.963227 23406 raft_consensus.cc:2781] T 9be73ad9ceec422492ce89080293be29 P 9a3cb170e246427c803511b585170a0f [term 1 FOLLOWER]: Leader election won for term 1
I20260620 06:08:29.963446 23406 raft_consensus.cc:686] T 9be73ad9ceec422492ce89080293be29 P 9a3cb170e246427c803511b585170a0f [term 1 LEADER]: Becoming Leader. State: Replica: 9a3cb170e246427c803511b585170a0f, State: Running, Role: LEADER
I20260620 06:08:29.963853 23406 consensus_queue.cc:237] T 9be73ad9ceec422492ce89080293be29 P 9a3cb170e246427c803511b585170a0f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3cb170e246427c803511b585170a0f" member_type: VOTER last_known_addr { host: "127.11.102.2" port: 38369 } } peers { permanent_uuid: "cbe512eba99e411d8948edb8a6d628ee" member_type: VOTER last_known_addr { host: "127.11.102.3" port: 41945 } } peers { permanent_uuid: "481b27dada144085a879c273ed236c08" member_type: VOTER last_known_addr { host: "127.11.102.1" port: 46267 } }
I20260620 06:08:29.966197 23410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:08:29.944324 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.102.3:41945 (local address 127.11.102.2:44159)
0620 06:08:29.950816 (+ 6492us) negotiation.cc:107] Waiting for socket to connect
0620 06:08:29.950833 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 06:08:29.950956 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:08:29.956977 (+ 6021us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:08:29.956980 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:08:29.957001 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 06:08:29.957124 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:08:29.957129 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:08:29.958157 (+ 1028us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:08:29.958163 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:08:29.960987 (+ 2824us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:08:29.960999 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:08:29.962077 (+ 1078us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:08:29.962080 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:08:29.962182 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:08:29.962206 (+ 24us) client_negotiation.cc:769] Sending connection context
0620 06:08:29.965966 (+ 3760us) client_negotiation.cc:240] Negotiation successful
0620 06:08:29.966052 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6419,"thread_start_us":102,"threads_started":1}
I20260620 06:08:29.967237 22896 catalog_manager.cc:5495] T 9be73ad9ceec422492ce89080293be29 P 9a3cb170e246427c803511b585170a0f reported cstate change: term changed from 0 to 1, leader changed from <none> to 9a3cb170e246427c803511b585170a0f (127.11.102.2). New cstate: current_term: 1 leader_uuid: "9a3cb170e246427c803511b585170a0f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3cb170e246427c803511b585170a0f" member_type: VOTER last_known_addr { host: "127.11.102.2" port: 38369 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cbe512eba99e411d8948edb8a6d628ee" member_type: VOTER last_known_addr { host: "127.11.102.3" port: 41945 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "481b27dada144085a879c273ed236c08" member_type: VOTER last_known_addr { host: "127.11.102.1" port: 46267 } health_report { overall_health: UNKNOWN } } }
I20260620 06:08:29.967955 23396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:08:29.950050 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.102.1:45675 (local address 127.11.102.3:41945)
0620 06:08:29.962741 (+ 12691us) server_negotiation.cc:206] Beginning negotiation
0620 06:08:29.962745 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 06:08:29.962758 (+ 13us) server_negotiation.cc:407] Connection header received
0620 06:08:29.962796 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:08:29.962799 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:08:29.962843 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:08:29.962912 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 06:08:29.963502 (+ 590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:08:29.964327 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:08:29.966527 (+ 2200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:08:29.967124 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:08:29.967215 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:08:29.967261 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0620 06:08:29.967764 (+ 503us) server_negotiation.cc:299] Negotiation successful
0620 06:08:29.967835 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12629,"thread_start_us":100,"threads_started":1}
I20260620 06:08:29.968551 23414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:08:29.949926 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.102.3:41945 (local address 127.11.102.1:45675)
0620 06:08:29.958486 (+ 8560us) negotiation.cc:107] Waiting for socket to connect
0620 06:08:29.958501 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 06:08:29.958587 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:08:29.963187 (+ 4600us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:08:29.963191 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:08:29.963210 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 06:08:29.963378 (+ 168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:08:29.963385 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:08:29.964535 (+ 1150us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:08:29.964541 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:08:29.965853 (+ 1312us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:08:29.965864 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:08:29.967373 (+ 1509us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:08:29.967377 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:08:29.967495 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:08:29.967518 (+ 23us) client_negotiation.cc:769] Sending connection context
0620 06:08:29.968331 (+ 813us) client_negotiation.cc:240] Negotiation successful
0620 06:08:29.968408 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8488,"mutex_wait_us":1,"thread_start_us":122,"threads_started":1}
I20260620 06:08:29.969331 23332 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9be73ad9ceec422492ce89080293be29" candidate_uuid: "481b27dada144085a879c273ed236c08" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cbe512eba99e411d8948edb8a6d628ee" is_pre_election: true
I20260620 06:08:29.969625 23332 raft_consensus.cc:2454] T 9be73ad9ceec422492ce89080293be29 P cbe512eba99e411d8948edb8a6d628ee [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 481b27dada144085a879c273ed236c08 in term 0.
I20260620 06:08:29.969952 23324 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9be73ad9ceec422492ce89080293be29" candidate_uuid: "9a3cb170e246427c803511b585170a0f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cbe512eba99e411d8948edb8a6d628ee" is_pre_election: true
I20260620 06:08:29.970052 23324 raft_consensus.cc:2454] T 9be73ad9ceec422492ce89080293be29 P cbe512eba99e411d8948edb8a6d628ee [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9a3cb170e246427c803511b585170a0f in term 0.
I20260620 06:08:29.970175 23332 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9be73ad9ceec422492ce89080293be29" candidate_uuid: "9a3cb170e246427c803511b585170a0f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cbe512eba99e411d8948edb8a6d628ee"
I20260620 06:08:29.970255 23393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:08:29.950562 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.102.1:53583 (local address 127.11.102.2:38369)
0620 06:08:29.950673 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0620 06:08:29.950678 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 06:08:29.965921 (+ 15243us) server_negotiation.cc:407] Connection header received
0620 06:08:29.966082 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:08:29.966085 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:08:29.966124 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:08:29.966166 (+ 42us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 06:08:29.966664 (+ 498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:08:29.967305 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:08:29.968814 (+ 1509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:08:29.969344 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:08:29.969428 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:08:29.969472 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0620 06:08:29.970061 (+ 589us) server_negotiation.cc:299] Negotiation successful
0620 06:08:29.970144 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260620 06:08:29.970283 23332 raft_consensus.cc:3037] T 9be73ad9ceec422492ce89080293be29 P cbe512eba99e411d8948edb8a6d628ee [term 0 FOLLOWER]: Advancing to term 1
I20260620 06:08:29.970675 22998 leader_election.cc:302] T 9be73ad9ceec422492ce89080293be29 P 481b27dada144085a879c273ed236c08 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 481b27dada144085a879c273ed236c08, cbe512eba99e411d8948edb8a6d628ee; no voters:
I20260620 06:08:29.970970 23407 raft_consensus.cc:2741] T 9be73ad9ceec422492ce89080293be29 P 481b27dada144085a879c273ed236c08 [term 1 FOLLOWER]: Leader pre-election decision vote started in defunct term 0: won
I20260620 06:08:29.970676 23412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:08:29.950471 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.102.2:38369 (local address 127.11.102.1:53583)
0620 06:08:29.965834 (+ 15363us) negotiation.cc:107] Waiting for socket to connect
0620 06:08:29.965849 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 06:08:29.965979 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:08:29.966279 (+ 300us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:08:29.966282 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:08:29.966298 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 06:08:29.966549 (+ 251us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:08:29.966558 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:08:29.967446 (+ 888us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:08:29.967449 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:08:29.968661 (+ 1212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:08:29.968671 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:08:29.969563 (+ 892us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:08:29.969566 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:08:29.969871 (+ 305us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:08:29.969894 (+ 23us) client_negotiation.cc:769] Sending connection context
0620 06:08:29.970414 (+ 520us) client_negotiation.cc:240] Negotiation successful
0620 06:08:29.970491 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15285,"mutex_wait_us":137,"thread_start_us":84,"threads_started":1}
I20260620 06:08:29.971558 23332 raft_consensus.cc:2454] T 9be73ad9ceec422492ce89080293be29 P cbe512eba99e411d8948edb8a6d628ee [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9a3cb170e246427c803511b585170a0f in term 1.
I20260620 06:08:29.971897 23200 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9be73ad9ceec422492ce89080293be29" candidate_uuid: "481b27dada144085a879c273ed236c08" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a3cb170e246427c803511b585170a0f" is_pre_election: true
W20260620 06:08:29.991580 23111 tablet.cc:2369] T 9be73ad9ceec422492ce89080293be29 P 481b27dada144085a879c273ed236c08: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 06:08:30.012257 23393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:08:30.007173 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51656 (local address 127.11.102.2:38369)
0620 06:08:30.007302 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0620 06:08:30.007306 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 06:08:30.007322 (+ 16us) server_negotiation.cc:407] Connection header received
0620 06:08:30.007465 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:08:30.007468 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:08:30.007532 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:08:30.007657 (+ 125us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 06:08:30.008027 (+ 370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:08:30.008836 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:08:30.010102 (+ 1266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:08:30.010444 (+ 342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:08:30.010534 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:08:30.010954 (+ 420us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 06:08:30.011505 (+ 551us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 06:08:30.012028 (+ 523us) server_negotiation.cc:1035] Waiting for connection context
0620 06:08:30.012062 (+ 34us) server_negotiation.cc:299] Negotiation successful
0620 06:08:30.012120 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260620 06:08:30.019569 23046 raft_consensus.cc:1261] T 9be73ad9ceec422492ce89080293be29 P 481b27dada144085a879c273ed236c08 [term 1 FOLLOWER]: Refusing update from remote peer 9a3cb170e246427c803511b585170a0f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 06:08:30.020310 23415 consensus_queue.cc:1035] T 9be73ad9ceec422492ce89080293be29 P 9a3cb170e246427c803511b585170a0f [LEADER]: Connected to new peer: Peer: permanent_uuid: "481b27dada144085a879c273ed236c08" member_type: VOTER last_known_addr { host: "127.11.102.1" port: 46267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 06:08:30.027995 23332 raft_consensus.cc:1261] T 9be73ad9ceec422492ce89080293be29 P cbe512eba99e411d8948edb8a6d628ee [term 1 FOLLOWER]: Refusing update from remote peer 9a3cb170e246427c803511b585170a0f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 06:08:30.028690 23415 consensus_queue.cc:1035] T 9be73ad9ceec422492ce89080293be29 P 9a3cb170e246427c803511b585170a0f [LEADER]: Connected to new peer: Peer: permanent_uuid: "cbe512eba99e411d8948edb8a6d628ee" member_type: VOTER last_known_addr { host: "127.11.102.3" port: 41945 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 06:08:30.040462 23418 mvcc.cc:205] Tried to move back new op lower bound from 7298808668231311360 to 7298808668015296512. Current Snapshot: MvccSnapshot[applied={T|T < 7298808668231311360}]
I20260620 06:08:30.049342 23421 mvcc.cc:205] Tried to move back new op lower bound from 7298808668231311360 to 7298808668015296512. Current Snapshot: MvccSnapshot[applied={T|T < 7298808668231311360}]
I20260620 06:08:30.049526 23419 mvcc.cc:205] Tried to move back new op lower bound from 7298808668231311360 to 7298808668015296512. Current Snapshot: MvccSnapshot[applied={T|T < 7298808668231311360}]
I20260620 06:08:30.060386 22896 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35380:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 06:08:30.060575 22896 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35380:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 06:08:30.063526 22896 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d7bfab7a8cf74c539a5ada4fe744c741: Sending DeleteTablet for 3 replicas of tablet 9be73ad9ceec422492ce89080293be29
I20260620 06:08:30.064546 23175 tablet_service.cc:1514] Processing DeleteTablet for tablet 9be73ad9ceec422492ce89080293be29 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 06:08:30 UTC) from {username='kudu', principal='kudu/127.11.102.62@KRBTEST.COM'} at 127.0.0.1:51654
I20260620 06:08:30.064677 23313 tablet_service.cc:1514] Processing DeleteTablet for tablet 9be73ad9ceec422492ce89080293be29 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 06:08:30 UTC) from {username='kudu', principal='kudu/127.11.102.62@KRBTEST.COM'} at 127.0.0.1:33578
I20260620 06:08:30.065300 23429 tablet_replica.cc:330] T 9be73ad9ceec422492ce89080293be29 P cbe512eba99e411d8948edb8a6d628ee: stopping tablet replica
I20260620 06:08:30.065485 23026 tablet_service.cc:1514] Processing DeleteTablet for tablet 9be73ad9ceec422492ce89080293be29 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 06:08:30 UTC) from {username='kudu', principal='kudu/127.11.102.62@KRBTEST.COM'} at 127.0.0.1:38204
I20260620 06:08:30.065879 23429 raft_consensus.cc:2229] T 9be73ad9ceec422492ce89080293be29 P cbe512eba99e411d8948edb8a6d628ee [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 06:08:30.066310 11672 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9HMp43/build/debug/bin/kudu with pid 22977
I20260620 06:08:30.066208 23429 raft_consensus.cc:2258] T 9be73ad9ceec422492ce89080293be29 P cbe512eba99e411d8948edb8a6d628ee [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 06:08:30.069077 23429 ts_tablet_manager.cc:1905] T 9be73ad9ceec422492ce89080293be29 P cbe512eba99e411d8948edb8a6d628ee: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 06:08:30.070642 23428 tablet_replica.cc:330] T 9be73ad9ceec422492ce89080293be29 P 9a3cb170e246427c803511b585170a0f: stopping tablet replica
I20260620 06:08:30.071033 23428 raft_consensus.cc:2229] T 9be73ad9ceec422492ce89080293be29 P 9a3cb170e246427c803511b585170a0f [term 1 LEADER]: Raft consensus shutting down.
I20260620 06:08:30.071498 23428 raft_consensus.cc:2258] T 9be73ad9ceec422492ce89080293be29 P 9a3cb170e246427c803511b585170a0f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 06:08:30.072038 23429 ts_tablet_manager.cc:1918] T 9be73ad9ceec422492ce89080293be29 P cbe512eba99e411d8948edb8a6d628ee: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 06:08:30.072129 23429 log.cc:1196] T 9be73ad9ceec422492ce89080293be29 P cbe512eba99e411d8948edb8a6d628ee: Deleting WAL directory at /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781935603120653-11672-0/minicluster-data/ts-2/wal/wals/9be73ad9ceec422492ce89080293be29
I20260620 06:08:30.072824 23429 ts_tablet_manager.cc:1939] T 9be73ad9ceec422492ce89080293be29 P cbe512eba99e411d8948edb8a6d628ee: Deleting consensus metadata
I20260620 06:08:30.073251 23428 ts_tablet_manager.cc:1905] T 9be73ad9ceec422492ce89080293be29 P 9a3cb170e246427c803511b585170a0f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 06:08:30.074069 22883 catalog_manager.cc:4841] TS cbe512eba99e411d8948edb8a6d628ee (127.11.102.3:41945): tablet 9be73ad9ceec422492ce89080293be29 (table test-table [id=8a8143bc6851402eb8e7b957aeafbccd]) successfully deleted
I20260620 06:08:30.076558 23428 ts_tablet_manager.cc:1918] T 9be73ad9ceec422492ce89080293be29 P 9a3cb170e246427c803511b585170a0f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 06:08:30.076691 23428 log.cc:1196] T 9be73ad9ceec422492ce89080293be29 P 9a3cb170e246427c803511b585170a0f: Deleting WAL directory at /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781935603120653-11672-0/minicluster-data/ts-1/wal/wals/9be73ad9ceec422492ce89080293be29
I20260620 06:08:30.077250 23428 ts_tablet_manager.cc:1939] T 9be73ad9ceec422492ce89080293be29 P 9a3cb170e246427c803511b585170a0f: Deleting consensus metadata
I20260620 06:08:30.078436 22880 catalog_manager.cc:4841] TS 9a3cb170e246427c803511b585170a0f (127.11.102.2:38369): tablet 9be73ad9ceec422492ce89080293be29 (table test-table [id=8a8143bc6851402eb8e7b957aeafbccd]) successfully deleted
W20260620 06:08:30.079587 22882 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.11.102.1:46267 (error 108)
I20260620 06:08:30.080549 11672 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9HMp43/build/debug/bin/kudu with pid 23116
I20260620 06:08:30.080641 23394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:08:30.080296 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.102.1:46267 (local address 127.0.0.1:38214)
0620 06:08:30.080492 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0620 06:08:30.080557 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.102.1:46267: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120}
W20260620 06:08:30.084988 22882 catalog_manager.cc:4584] TS 481b27dada144085a879c273ed236c08 (127.11.102.1:46267): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 9be73ad9ceec422492ce89080293be29: Network error: Client connection negotiation failed: client connection to 127.11.102.1:46267: connect: Connection refused (error 111)
I20260620 06:08:30.092025 11672 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9HMp43/build/debug/bin/kudu with pid 23251
I20260620 06:08:30.107206 11672 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9HMp43/build/debug/bin/kudu with pid 22864
I20260620 06:08:30.116456 11672 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260620 06:08:30.461189 11672 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260620 06:08:30.461326 11672 mini_ranger.cc:66] Stopping Ranger...
I20260620 06:08:30.806012 11672 mini_ranger.cc:68] Stopped Ranger
2026-06-20 06:08:30.806 UTC [22450] LOG: received smart shutdown request
2026-06-20 06:08:30.809 UTC [22450] LOG: background worker "logical replication launcher" (PID 22459) exited with exit code 1
2026-06-20 06:08:30.809 UTC [22454] LOG: shutting down
2026-06-20 06:08:30.969 UTC [22450] LOG: database system is shut down
2026-06-20T06:08:30Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (46436 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935603120653-11672-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 06:08:31 dist-test-slave-9hm5 krb5kdc[23440](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 06:08:31 dist-test-slave-9hm5 krb5kdc[23440](info): set up 2 sockets
Jun 20 06:08:31 dist-test-slave-9hm5 krb5kdc[23440](info): commencing operation
krb5kdc: starting...
W20260620 06:08:32.123690 11672 mini_kdc.cc:121] Time spent starting KDC: real 1.015s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935603120653-11672-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935603120653-11672-0/krb5kdc/test-user.keytab.
Jun 20 06:08:32 dist-test-slave-9hm5 krb5kdc[23440](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935712, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T06:08:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T06:08:32Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.11.102.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.11.102.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.11.102.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935603120653-11672-0/krb5kdc/rangeradmin_127.11.102.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.11.102.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935603120653-11672-0/krb5kdc/rangeradmin_127.11.102.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.11.102.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.11.102.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.11.102.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935603120653-11672-0/krb5kdc/rangerlookup_127.11.102.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.11.102.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935603120653-11672-0/krb5kdc/rangerlookup_127.11.102.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.11.102.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.11.102.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.11.102.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935603120653-11672-0/krb5kdc/HTTP_127.11.102.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.11.102.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935603120653-11672-0/krb5kdc/HTTP_127.11.102.20@KRBTEST.COM.keytab.
I20260620 06:08:32.279384 11672 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935603120653-11672-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-task9HMp43/build/debug/bin/postgres/pg_ctl -D /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935603120653-11672-0/minicluster-data/postgres -l logfile start
2026-06-20 06:08:34.662 UTC [23470] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-06-20 06:08:34.662 UTC [23470] LOG: listening on IPv4 address "127.11.102.20", port 51443
2026-06-20 06:08:34.666 UTC [23470] LOG: listening on Unix socket "/tmp/.s.PGSQL.51443"
2026-06-20 06:08:34.699 UTC [23473] LOG: database system was shut down at 2026-06-20 06:08:33 UTC
2026-06-20 06:08:34.703 UTC [23470] LOG: database system is ready to accept connections
I20260620 06:08:35.659489 11672 mini_postgres.cc:96] Postgres bound to 51443
2026-06-20 06:08:35.665 UTC [23481] FATAL: database "slave" does not exist
127.11.102.20:51443 - accepting connections
I20260620 06:08:35.666251 11672 mini_ranger.cc:161] Starting Ranger...
I20260620 06:08:35.681679 11672 mini_ranger.cc:84] Created miniranger Postgres user
I20260620 06:08:36.120628 11672 mini_ranger.cc:87] Created ranger Postgres database
I20260620 06:08:36.120723 11672 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935603120653-11672-0/minicluster-data/ranger-admin
2026-06-20 06:08:36,418 [I] DB FLAVOR :POSTGRES
2026-06-20 06:08:36,418 [I] --------- Verifying Ranger DB connection ---------
2026-06-20 06:08:36,418 [I] Checking connection..
2026-06-20 06:08:36,418 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task9HMp43/build/debug/bin/postgresql.jar:/tmp/dist-test-task9HMp43/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.102.20:51443/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-06-20 06:08:36,669 [I] Checking connection passed.
2026-06-20 06:08:36,669 [I] --------- Verifying version history table ---------
2026-06-20 06:08:36,670 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task9HMp43/build/debug/bin/postgresql.jar:/tmp/dist-test-task9HMp43/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.102.20:51443/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-20 06:08:36,892 [I] Table x_db_version_h does not exist in database ranger
2026-06-20 06:08:36,892 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task9HMp43/build/debug/bin/postgresql.jar:/tmp/dist-test-task9HMp43/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.102.20:51443/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-20 06:08:37,123 [I] Table x_db_version_h does not exist in database ranger
2026-06-20 06:08:37,123 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-06-20 06:08:37,124 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task9HMp43/build/debug/bin/postgresql.jar:/tmp/dist-test-task9HMp43/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.102.20:51443/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task9HMp43/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-06-20 06:08:37.367 UTC [23545] WARNING: there is no transaction in progress
2026-06-20 06:08:37,372 [I] create_dbversion_catalog.sql file imported successfully
2026-06-20 06:08:37,373 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task9HMp43/build/debug/bin/postgresql.jar:/tmp/dist-test-task9HMp43/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.102.20:51443/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-20 06:08:37,626 [I] Table x_db_version_h already exists in database 'ranger'
2026-06-20 06:08:37,626 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-20 06:08:37,626 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task9HMp43/build/debug/bin/postgresql.jar:/tmp/dist-test-task9HMp43/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.102.20:51443/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-06-20 06:08:37,886 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task9HMp43/build/debug/bin/postgresql.jar:/tmp/dist-test-task9HMp43/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.102.20:51443/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-06-20 06:08:38,134 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task9HMp43/build/debug/bin/postgresql.jar:/tmp/dist-test-task9HMp43/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.102.20:51443/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-9hm5.c.gcp-upstream.internal','N') ;"
2026-06-20 06:08:38,359 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task9HMp43/build/debug/bin/postgresql.jar:/tmp/dist-test-task9HMp43/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.102.20:51443/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-20 06:08:38,583 [I] Table x_portal_user does not exist in database ranger
2026-06-20 06:08:38,583 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task9HMp43/build/debug/bin/postgresql.jar:/tmp/dist-test-task9HMp43/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.102.20:51443/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-20 06:08:38,823 [I] Table x_policy_ref_group does not exist in database ranger
2026-06-20 06:08:38,823 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task9HMp43/build/debug/bin/postgresql.jar:/tmp/dist-test-task9HMp43/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.102.20:51443/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-20 06:08:39,077 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task9HMp43/build/debug/bin/postgresql.jar:/tmp/dist-test-task9HMp43/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.102.20:51443/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-20 06:08:39,333 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-06-20 06:08:39,333 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task9HMp43/build/debug/bin/postgresql.jar:/tmp/dist-test-task9HMp43/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.102.20:51443/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task9HMp43/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-06-20 06:08:40.308 UTC [23653] WARNING: there is no transaction in progress
2026-06-20 06:08:40.324 UTC [23653] WARNING: there is no transaction in progress
2026-06-20 06:08:40.339 UTC [23653] WARNING: there is no transaction in progress
2026-06-20 06:08:40.354 UTC [23653] WARNING: there is no transaction in progress
2026-06-20 06:08:40.370 UTC [23653] WARNING: there is no transaction in progress
2026-06-20 06:08:40.388 UTC [23653] WARNING: there is no transaction in progress
2026-06-20 06:08:40.443 UTC [23653] WARNING: there is no transaction in progress
2026-06-20 06:08:40.449 UTC [23653] WARNING: there is no transaction in progress
2026-06-20 06:08:40.464 UTC [23653] WARNING: there is no transaction in progress
2026-06-20 06:08:40.474 UTC [23653] WARNING: there is no transaction in progress
2026-06-20 06:08:40.483 UTC [23653] WARNING: there is no transaction in progress
2026-06-20 06:08:40.496 UTC [23653] WARNING: there is no transaction in progress
2026-06-20 06:08:40.506 UTC [23653] WARNING: there is no transaction in progress
2026-06-20 06:08:40.512 UTC [23653] WARNING: there is no transaction in progress
2026-06-20 06:08:40.518 UTC [23653] WARNING: there is no transaction in progress
2026-06-20 06:08:40.525 UTC [23653] WARNING: there is no transaction in progress
2026-06-20 06:08:41,132 [I] ranger_core_db_postgres.sql file imported successfully
2026-06-20 06:08:41,132 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task9HMp43/build/debug/bin/postgresql.jar:/tmp/dist-test-task9HMp43/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.102.20:51443/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-06-20 06:08:41,361 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-06-20 06:08:41,361 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task9HMp43/build/debug/bin/postgresql.jar:/tmp/dist-test-task9HMp43/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.102.20:51443/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-20 06:08:41,607 [I] Table x_portal_user already exists in database 'ranger'
2026-06-20 06:08:41,608 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task9HMp43/build/debug/bin/postgresql.jar:/tmp/dist-test-task9HMp43/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.102.20:51443/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-20 06:08:41,864 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-06-20 06:08:41,864 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task9HMp43/build/debug/bin/postgresql.jar:/tmp/dist-test-task9HMp43/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.102.20:51443/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-20 06:08:42,102 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task9HMp43/build/debug/bin/postgresql.jar:/tmp/dist-test-task9HMp43/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.102.20:51443/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-20 06:08:42,334 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task9HMp43/build/debug/bin/postgresql.jar:/tmp/dist-test-task9HMp43/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.102.20:51443/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-9hm5.c.gcp-upstream.internal';"
2026-06-20 06:08:42,566 [I] CORE_DB_SCHEMA import status has been updated
2026-06-20 06:08:42,567 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task9HMp43/build/debug/bin/postgresql.jar:/tmp/dist-test-task9HMp43/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.102.20:51443/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-06-20 06:08:42,824 [I] DB_PATCHES have already been applied
I20260620 06:08:42.829648 11672 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935603120653-11672-0/minicluster-data/ranger-admin:/tmp/dist-test-task9HMp43/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-task9HMp43/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-task9HMp43/build/debug/bin/postgresql.jar:/tmp/dist-test-task9HMp43/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260620 06:08:42.829712 11672 mini_ranger.cc:193] Using host: 127.11.102.20
I20260620 06:08:42.833077 11672 mini_ranger.cc:227] Ranger admin URL: http://127.11.102.20:41549
Jun 20, 2026 6:08:43 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 20, 2026 6:08:43 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 20, 2026 6:08:43 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935603120653-11672-0/minicluster-data/ranger-admin/ews/webapp
Jun 20, 2026 6:08:43 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935603120653-11672-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Jun 20, 2026 6:08:43 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935603120653-11672-0/minicluster-data/ranger-admin/ews/webapp] .....
Jun 20, 2026 6:08:43 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935603120653-11672-0/minicluster-data/ranger-admin/ews/webapp].
Jun 20, 2026 6:08:43 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260620 06:08:43.937813 11672 mini_ranger.cc:160] Time spent starting Ranger: real 8.272s user 0.004s sys 0.006s
Jun 20, 2026 6:09:02 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Jun 20, 2026 6:09:03 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
Jun 20, 2026 6:09:03 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Jun 20, 2026 6:09:03 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Jun 20, 2026 6:09:03 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Jun 20, 2026 6:09:03 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Jun 20, 2026 6:09:03 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Jun 20, 2026 6:09:03 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Jun 20, 2026 6:09:03 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Jun 20, 2026 6:09:03 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Jun 20, 2026 6:09:03 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Jun 20, 2026 6:09:03 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Jun 20, 2026 6:09:03 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Jun 20, 2026 6:09:03 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Jun 20, 2026 6:09:03 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Jun 20, 2026 6:09:03 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Jun 20, 2026 6:09:03 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Jun 20, 2026 6:09:03 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260620 06:09:04.525283 11672 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.11.102.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.11.102.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.11.102.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935603120653-11672-0/krb5kdc/rangerkms_127.11.102.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.11.102.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935603120653-11672-0/krb5kdc/rangerkms_127.11.102.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.11.102.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.11.102.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.11.102.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935603120653-11672-0/krb5kdc/HTTP_127.11.102.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.11.102.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935603120653-11672-0/krb5kdc/HTTP_127.11.102.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Jun 20 06:09:04 dist-test-slave-9hm5 krb5kdc[23440](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935744, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260620 06:09:04.606808 11672 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260620 06:09:04.616127 11672 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260620 06:09:05.385103 11672 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260620 06:09:05.385175 11672 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935603120653-11672-0/minicluster-data/ranger-kms
I20260620 06:09:05.385188 11672 mini_ranger_kms.cc:227] Using postgres at 127.11.102.20:51443
2026-06-20 06:09:05,551 [I] DB FLAVOR :POSTGRES
2026-06-20 06:09:05,552 [I] --------- Verifying Ranger DB connection ---------
2026-06-20 06:09:05,552 [I] Checking connection
2026-06-20 06:09:05,552 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task9HMp43/build/debug/bin/postgresql.jar:/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935603120653-11672-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.102.20:51443/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-06-20 06:09:05,789 [I] connection success
2026-06-20 06:09:05,790 [I] --------- Verifying Ranger DB tables ---------
2026-06-20 06:09:05,790 [I] Verifying table ranger_masterkey in database rangerkms
2026-06-20 06:09:05,790 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task9HMp43/build/debug/bin/postgresql.jar:/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935603120653-11672-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.102.20:51443/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-06-20 06:09:06,014 [I] Table ranger_masterkey does not exist in database rangerkms
2026-06-20 06:09:06,015 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-20 06:09:06,015 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-06-20 06:09:06,015 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task9HMp43/build/debug/bin/postgresql.jar:/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935603120653-11672-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.11.102.20:51443/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935603120653-11672-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-06-20 06:09:06,258 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260620 06:09:06.700217 11672 mini_ranger_kms.cc:310] Created kms service
I20260620 06:09:06.876633 11672 mini_ranger_kms.cc:326] Created kudu user
I20260620 06:09:06.917557 11672 mini_ranger_kms.cc:343] Created rangerkms user
Jun 20 06:09:06 dist-test-slave-9hm5 krb5kdc[23440](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935744, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.11.102.20@KRBTEST.COM
I20260620 06:09:07.281194 11672 mini_ranger_kms.cc:384] Added ranger policy
I20260620 06:09:07.281351 11672 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935603120653-11672-0/minicluster-data/ranger-kms:/tmp/dist-test-task9HMp43/build/debug/bin/postgresql.jar:/tmp/dist-test-task9HMp43/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-task9HMp43/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260620 06:09:07.281383 11672 mini_ranger_kms.cc:240] Using host: 127.11.102.20
I20260620 06:09:07.285035 11672 mini_ranger_kms.cc:276] Ranger KMS PID: 23833
I20260620 06:09:07.285116 11672 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.11.102.20:53659
Jun 20, 2026 6:09:08 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 20, 2026 6:09:08 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 20, 2026 6:09:08 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935603120653-11672-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Jun 20, 2026 6:09:08 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935603120653-11672-0/minicluster-data/ranger-kms/ews/webapp] .....
Jun 20, 2026 6:09:08 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935603120653-11672-0/minicluster-data/ranger-kms/ews/webapp].
Jun 20, 2026 6:09:08 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260620 06:09:09.452072 11672 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 4.845s user 0.018s sys 0.131s
I20260620 06:09:09.452201 11672 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260620 06:09:09.452309 11672 mini_ranger_kms.cc:401] 127.11.102.20:53659/kms/v1/keys
Jun 20 06:09:09 dist-test-slave-9hm5 krb5kdc[23440](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781935749, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.11.102.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Jun 20 06:09:09 dist-test-slave-9hm5 krb5kdc[23440](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935749, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.11.102.20@KRBTEST.COM for HTTP/127.11.102.20@KRBTEST.COM
Jun 20 06:09:10 dist-test-slave-9hm5 krb5kdc[23440](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935749, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.11.102.20@KRBTEST.COM for HTTP/127.11.102.20@KRBTEST.COM
Jun 20 06:09:12 dist-test-slave-9hm5 krb5kdc[23440](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935752, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.11.102.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.102.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.102.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935603120653-11672-0/krb5kdc/kudu_127.11.102.62.keytab.
Entry for principal kudu/127.11.102.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935603120653-11672-0/krb5kdc/kudu_127.11.102.62.keytab.
I20260620 06:09:12.550585 11672 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9HMp43/build/debug/bin/kudu
/tmp/dist-test-task9HMp43/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935603120653-11672-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935603120653-11672-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935603120653-11672-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935603120653-11672-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.102.62:40839
--webserver_interface=127.11.102.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935603120653-11672-0/krb5kdc/kudu_127.11.102.62.keytab
--principal=kudu/127.11.102.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.102.20:46459
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.102.62:40839
--ranger_config_path=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935603120653-11672-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.11.102.20:53659/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935603120653-11672-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935603120653-11672-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935603120653-11672-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 06:09:12.661660 23884 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:09:12.661965 23884 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:09:12.662035 23884 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:09:12.665997 23884 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 06:09:12.666085 23884 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:09:12.666123 23884 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:09:12.666180 23884 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 06:09:12.666220 23884 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 06:09:12.671559 23884 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.102.20:46459
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935603120653-11672-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935603120653-11672-0/minicluster-data/master-0/wal
--ranger_kms_url=127.11.102.20:53659/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.11.102.62:40839
--ranger_config_path=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935603120653-11672-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935603120653-11672-0/krb5kdc/kudu_127.11.102.62.keytab
--principal=kudu/127.11.102.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.102.62:40839
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935603120653-11672-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.102.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23884
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935603120653-11672-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 05:28:48 UTC on ccycloud.quasar-fixhms.root.comops.site
build id 4457020
I20260620 06:09:12.672670 23884 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:09:12.673741 23884 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 06:09:12.682585 23892 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:09:12.682626 23889 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:09:12.682623 23884 server_base.cc:1034] running on GCE node
W20260620 06:09:12.682585 23890 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:09:12.683445 23884 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:09:12.684595 23884 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:09:12.685778 23884 hybrid_clock.cc:648] HybridClock initialized: now 1781935752685762 us; error 31 us; skew 500 ppm
Jun 20 06:09:12 dist-test-slave-9hm5 krb5kdc[23440](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935752, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.102.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 06:09:12.688516 23884 init.cc:377] Logged in from keytab as kudu/127.11.102.62@KRBTEST.COM (short username kudu)
I20260620 06:09:12.689345 23884 webserver.cc:460] Webserver started at http://127.11.102.62:42523/ using document root <none> and password file <none>
I20260620 06:09:12.689822 23884 fs_manager.cc:359] Metadata directory not provided
I20260620 06:09:12.689872 23884 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:09:12.690039 23884 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 06:09:12 dist-test-slave-9hm5 krb5kdc[23440](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935752, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.102.62@KRBTEST.COM for HTTP/127.11.102.20@KRBTEST.COM
I20260620 06:09:12.746380 23884 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935603120653-11672-0/minicluster-data/master-0/data/instance:
uuid: "f42b7083570d4a95b735450329fb1c2f"
format_stamp: "Formatted at 2026-06-20 06:09:12 on dist-test-slave-9hm5"
server_key: "be182a1c82fc83eac5806eb34b42e7a1"
server_key_iv: "ffb27ff7200324dce2a19b4e79fde437"
server_key_version: "kuduclusterkey@0"
I20260620 06:09:12.746996 23884 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935603120653-11672-0/minicluster-data/master-0/wal/instance:
uuid: "f42b7083570d4a95b735450329fb1c2f"
format_stamp: "Formatted at 2026-06-20 06:09:12 on dist-test-slave-9hm5"
server_key: "be182a1c82fc83eac5806eb34b42e7a1"
server_key_iv: "ffb27ff7200324dce2a19b4e79fde437"
server_key_version: "kuduclusterkey@0"
I20260620 06:09:12.750891 23884 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.001s
I20260620 06:09:12.767604 23899 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:09:12.769083 23884 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 06:09:12.769227 23884 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935603120653-11672-0/minicluster-data/master-0/data,/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935603120653-11672-0/minicluster-data/master-0/wal
uuid: "f42b7083570d4a95b735450329fb1c2f"
format_stamp: "Formatted at 2026-06-20 06:09:12 on dist-test-slave-9hm5"
server_key: "be182a1c82fc83eac5806eb34b42e7a1"
server_key_iv: "ffb27ff7200324dce2a19b4e79fde437"
server_key_version: "kuduclusterkey@0"
I20260620 06:09:12.769363 23884 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935603120653-11672-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935603120653-11672-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935603120653-11672-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:09:12.779341 23884 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:09:12.782148 23884 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:09:12.782336 23884 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:09:12.789448 23884 rpc_server.cc:307] RPC server started. Bound to: 127.11.102.62:40839
I20260620 06:09:12.789469 23951 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.102.62:40839 every 8 connection(s)
I20260620 06:09:12.790376 23884 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935603120653-11672-0/minicluster-data/master-0/data/info.pb
I20260620 06:09:12.793183 23952 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 06:09:12.797911 11672 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9HMp43/build/debug/bin/kudu as pid 23884
I20260620 06:09:12.798063 11672 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935603120653-11672-0/minicluster-data/master-0/wal/instance
I20260620 06:09:12.798338 11672 external_mini_cluster.cc:1428] Setting key 94320036a8d6a9c0efaa44996168cd8b
I20260620 06:09:12.798710 23952 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f42b7083570d4a95b735450329fb1c2f: Bootstrap starting.
I20260620 06:09:12.801127 23952 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f42b7083570d4a95b735450329fb1c2f: Neither blocks nor log segments found. Creating new log.
I20260620 06:09:12.802052 23952 log.cc:824] T 00000000000000000000000000000000 P f42b7083570d4a95b735450329fb1c2f: Log is configured to *not* fsync() on all Append() calls
I20260620 06:09:12.804124 23952 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f42b7083570d4a95b735450329fb1c2f: No bootstrap required, opened a new log
Jun 20 06:09:12 dist-test-slave-9hm5 krb5kdc[23440](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935752, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.102.62@KRBTEST.COM
I20260620 06:09:12.808621 23952 raft_consensus.cc:348] T 00000000000000000000000000000000 P f42b7083570d4a95b735450329fb1c2f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f42b7083570d4a95b735450329fb1c2f" member_type: VOTER last_known_addr { host: "127.11.102.62" port: 40839 } }
I20260620 06:09:12.808923 23952 raft_consensus.cc:374] T 00000000000000000000000000000000 P f42b7083570d4a95b735450329fb1c2f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 06:09:12.809000 23952 raft_consensus.cc:729] T 00000000000000000000000000000000 P f42b7083570d4a95b735450329fb1c2f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f42b7083570d4a95b735450329fb1c2f, State: Initialized, Role: FOLLOWER
I20260620 06:09:12.809516 23952 consensus_queue.cc:260] T 00000000000000000000000000000000 P f42b7083570d4a95b735450329fb1c2f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f42b7083570d4a95b735450329fb1c2f" member_type: VOTER last_known_addr { host: "127.11.102.62" port: 40839 } }
I20260620 06:09:12.809676 23952 raft_consensus.cc:388] T 00000000000000000000000000000000 P f42b7083570d4a95b735450329fb1c2f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 06:09:12.809746 23952 raft_consensus.cc:482] T 00000000000000000000000000000000 P f42b7083570d4a95b735450329fb1c2f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 06:09:12.809847 23952 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f42b7083570d4a95b735450329fb1c2f [term 0 FOLLOWER]: Advancing to term 1
I20260620 06:09:12.810961 23952 raft_consensus.cc:504] T 00000000000000000000000000000000 P f42b7083570d4a95b735450329fb1c2f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f42b7083570d4a95b735450329fb1c2f" member_type: VOTER last_known_addr { host: "127.11.102.62" port: 40839 } }
I20260620 06:09:12.811317 23952 leader_election.cc:302] T 00000000000000000000000000000000 P f42b7083570d4a95b735450329fb1c2f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f42b7083570d4a95b735450329fb1c2f; no voters:
I20260620 06:09:12.811666 23952 leader_election.cc:288] T 00000000000000000000000000000000 P f42b7083570d4a95b735450329fb1c2f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 06:09:12.811757 23957 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f42b7083570d4a95b735450329fb1c2f [term 1 FOLLOWER]: Leader election won for term 1
I20260620 06:09:12.811998 23957 raft_consensus.cc:686] T 00000000000000000000000000000000 P f42b7083570d4a95b735450329fb1c2f [term 1 LEADER]: Becoming Leader. State: Replica: f42b7083570d4a95b735450329fb1c2f, State: Running, Role: LEADER
I20260620 06:09:12.812505 23957 consensus_queue.cc:237] T 00000000000000000000000000000000 P f42b7083570d4a95b735450329fb1c2f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f42b7083570d4a95b735450329fb1c2f" member_type: VOTER last_known_addr { host: "127.11.102.62" port: 40839 } }
I20260620 06:09:12.812964 23952 sys_catalog.cc:564] T 00000000000000000000000000000000 P f42b7083570d4a95b735450329fb1c2f [sys.catalog]: configured and running, proceeding with master startup.
I20260620 06:09:12.813504 23955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:09:12.799776 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55204 (local address 127.11.102.62:40839)
0620 06:09:12.800200 (+ 424us) server_negotiation.cc:206] Beginning negotiation
0620 06:09:12.800216 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0620 06:09:12.800251 (+ 35us) server_negotiation.cc:407] Connection header received
0620 06:09:12.800856 (+ 605us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:09:12.800873 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:09:12.801198 (+ 325us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:09:12.801537 (+ 339us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:09:12.802462 (+ 925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:09:12.803553 (+ 1091us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:09:12.804713 (+ 1160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:09:12.805167 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:09:12.805293 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:09:12.808113 (+ 2820us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:09:12.808119 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:09:12.808136 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:09:12.808166 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:09:12.810042 (+ 1876us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:09:12.810459 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:09:12.810465 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:09:12.810471 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:09:12.810603 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:09:12.810856 (+ 253us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:09:12.810859 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:09:12.810860 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:09:12.811361 (+ 501us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:09:12.811565 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0620 06:09:12.812294 (+ 729us) server_negotiation.cc:299] Negotiation successful
0620 06:09:12.812571 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":284,"thread_start_us":143,"threads_started":1}
I20260620 06:09:12.814162 23959 sys_catalog.cc:455] T 00000000000000000000000000000000 P f42b7083570d4a95b735450329fb1c2f [sys.catalog]: SysCatalogTable state changed. Reason: New leader f42b7083570d4a95b735450329fb1c2f. Latest consensus state: current_term: 1 leader_uuid: "f42b7083570d4a95b735450329fb1c2f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f42b7083570d4a95b735450329fb1c2f" member_type: VOTER last_known_addr { host: "127.11.102.62" port: 40839 } } }
I20260620 06:09:12.814276 23959 sys_catalog.cc:458] T 00000000000000000000000000000000 P f42b7083570d4a95b735450329fb1c2f [sys.catalog]: This master's current role is: LEADER
I20260620 06:09:12.814597 23958 sys_catalog.cc:455] T 00000000000000000000000000000000 P f42b7083570d4a95b735450329fb1c2f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f42b7083570d4a95b735450329fb1c2f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f42b7083570d4a95b735450329fb1c2f" member_type: VOTER last_known_addr { host: "127.11.102.62" port: 40839 } } }
I20260620 06:09:12.814667 23958 sys_catalog.cc:458] T 00000000000000000000000000000000 P f42b7083570d4a95b735450329fb1c2f [sys.catalog]: This master's current role is: LEADER
I20260620 06:09:12.815903 23967 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 06:09:12.818360 23967 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 06:09:12.818851 23952 ranger_client.cc:318] Using new properties file: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935603120653-11672-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260620 06:09:12.827970 23967 catalog_manager.cc:1347] Generated new cluster ID: b64f44e878a44f049260905de020040a
I20260620 06:09:12.828102 23967 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 06:09:12.836966 23967 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 06:09:12.837931 23967 catalog_manager.cc:1504] Loading token signing keys...
I20260620 06:09:12.846443 23967 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f42b7083570d4a95b735450329fb1c2f: Generated new TSK 0
I20260620 06:09:12.847204 23967 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.11.102.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.102.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.102.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935603120653-11672-0/krb5kdc/kudu_127.11.102.1.keytab.
Entry for principal kudu/127.11.102.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935603120653-11672-0/krb5kdc/kudu_127.11.102.1.keytab.
I20260620 06:09:13.502357 11672 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9HMp43/build/debug/bin/kudu
/tmp/dist-test-task9HMp43/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935603120653-11672-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935603120653-11672-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935603120653-11672-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935603120653-11672-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.11.102.20:53659/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.102.1:0
--local_ip_for_outbound_sockets=127.11.102.1
--webserver_interface=127.11.102.1
--webserver_port=0
--tserver_master_addrs=127.11.102.62:40839
--keytab_file=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935603120653-11672-0/krb5kdc/kudu_127.11.102.1.keytab
--principal=kudu/127.11.102.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.102.20:46459
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935603120653-11672-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935603120653-11672-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935603120653-11672-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 06:09:13.651579 23997 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:09:13.651966 23997 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:09:13.652037 23997 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:09:13.657296 23997 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:09:13.657356 23997 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:09:13.657436 23997 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.102.1
I20260620 06:09:13.664179 23997 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.102.20:46459
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935603120653-11672-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935603120653-11672-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.11.102.20:53659/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935603120653-11672-0/krb5kdc/kudu_127.11.102.1.keytab
--principal=kudu/127.11.102.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.102.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935603120653-11672-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.11.102.1
--webserver_port=0
--tserver_master_addrs=127.11.102.62:40839
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23997
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.102.1
--log_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935603120653-11672-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 05:28:48 UTC on ccycloud.quasar-fixhms.root.comops.site
build id 4457020
I20260620 06:09:13.665823 23997 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:09:13.667167 23997 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 06:09:13.678285 24005 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:09:13.678607 23997 server_base.cc:1034] running on GCE node
W20260620 06:09:13.678078 24004 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:09:13.678085 24007 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:09:13.679289 23997 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:09:13.680073 23997 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:09:13.681313 23997 hybrid_clock.cc:648] HybridClock initialized: now 1781935753681295 us; error 43 us; skew 500 ppm
Jun 20 06:09:13 dist-test-slave-9hm5 krb5kdc[23440](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935753, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.102.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 06:09:13.684906 23997 init.cc:377] Logged in from keytab as kudu/127.11.102.1@KRBTEST.COM (short username kudu)
I20260620 06:09:13.685935 23997 webserver.cc:460] Webserver started at http://127.11.102.1:46761/ using document root <none> and password file <none>
I20260620 06:09:13.686611 23997 fs_manager.cc:359] Metadata directory not provided
I20260620 06:09:13.686702 23997 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:09:13.686931 23997 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 06:09:13 dist-test-slave-9hm5 krb5kdc[23440](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935753, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.102.1@KRBTEST.COM for HTTP/127.11.102.20@KRBTEST.COM
I20260620 06:09:13.702363 23997 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935603120653-11672-0/minicluster-data/ts-0/data/instance:
uuid: "c3a5167af887498198f016f81be7103f"
format_stamp: "Formatted at 2026-06-20 06:09:13 on dist-test-slave-9hm5"
server_key: "5f35d8f9e27a5f28f37b2db7dac0250e"
server_key_iv: "b30c368c808cea2ced7a871640e5eb46"
server_key_version: "kuduclusterkey@0"
I20260620 06:09:13.702984 23997 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935603120653-11672-0/minicluster-data/ts-0/wal/instance:
uuid: "c3a5167af887498198f016f81be7103f"
format_stamp: "Formatted at 2026-06-20 06:09:13 on dist-test-slave-9hm5"
server_key: "5f35d8f9e27a5f28f37b2db7dac0250e"
server_key_iv: "b30c368c808cea2ced7a871640e5eb46"
server_key_version: "kuduclusterkey@0"
I20260620 06:09:13.706948 23997 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260620 06:09:13.728086 24014 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:09:13.729897 23997 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260620 06:09:13.730038 23997 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935603120653-11672-0/minicluster-data/ts-0/data,/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935603120653-11672-0/minicluster-data/ts-0/wal
uuid: "c3a5167af887498198f016f81be7103f"
format_stamp: "Formatted at 2026-06-20 06:09:13 on dist-test-slave-9hm5"
server_key: "5f35d8f9e27a5f28f37b2db7dac0250e"
server_key_iv: "b30c368c808cea2ced7a871640e5eb46"
server_key_version: "kuduclusterkey@0"
I20260620 06:09:13.730170 23997 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935603120653-11672-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935603120653-11672-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935603120653-11672-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:09:13.786336 23997 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:09:13.797629 23997 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:09:13.797894 23997 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:09:13.798605 23997 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 06:09:13.799690 23997 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 06:09:13.799774 23997 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 06:09:13.799851 23997 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 06:09:13.799899 23997 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 06:09:13.810491 23997 rpc_server.cc:307] RPC server started. Bound to: 127.11.102.1:33233
I20260620 06:09:13.810519 24129 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.102.1:33233 every 8 connection(s)
I20260620 06:09:13.811487 23997 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935603120653-11672-0/minicluster-data/ts-0/data/info.pb
I20260620 06:09:13.819717 11672 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9HMp43/build/debug/bin/kudu as pid 23997
I20260620 06:09:13.819840 11672 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935603120653-11672-0/minicluster-data/ts-0/wal/instance
I20260620 06:09:13.820175 11672 external_mini_cluster.cc:1428] Setting key 751ff2d3c8507502d951079df0ea0f24
Jun 20 06:09:13 dist-test-slave-9hm5 krb5kdc[23440](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935753, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.102.1@KRBTEST.COM for kudu/127.11.102.62@KRBTEST.COM
I20260620 06:09:13.826370 24132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:09:13.813468 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.102.1:54235 (local address 127.11.102.62:40839)
0620 06:09:13.813732 (+ 264us) server_negotiation.cc:206] Beginning negotiation
0620 06:09:13.813741 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 06:09:13.815088 (+ 1347us) server_negotiation.cc:407] Connection header received
0620 06:09:13.816043 (+ 955us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:09:13.816046 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:09:13.816102 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:09:13.816223 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:09:13.817358 (+ 1135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:09:13.817922 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:09:13.819094 (+ 1172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:09:13.819335 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:09:13.820860 (+ 1525us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:09:13.822213 (+ 1353us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:09:13.822219 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:09:13.822223 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:09:13.822251 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:09:13.824066 (+ 1815us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:09:13.824708 (+ 642us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:09:13.824714 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:09:13.824717 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:09:13.824810 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:09:13.825204 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:09:13.825209 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:09:13.825212 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:09:13.825473 (+ 261us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:09:13.825602 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0620 06:09:13.826071 (+ 469us) server_negotiation.cc:299] Negotiation successful
0620 06:09:13.826188 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":181,"thread_start_us":97,"threads_started":1}
I20260620 06:09:13.827276 24133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:09:13.813949 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.102.62:40839 (local address 127.11.102.1:54235)
0620 06:09:13.814948 (+ 999us) negotiation.cc:107] Waiting for socket to connect
0620 06:09:13.814987 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0620 06:09:13.815840 (+ 853us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:09:13.816385 (+ 545us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:09:13.816392 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:09:13.816722 (+ 330us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:09:13.817221 (+ 499us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:09:13.817233 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:09:13.818049 (+ 816us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:09:13.818053 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:09:13.818964 (+ 911us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:09:13.818984 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:09:13.819485 (+ 501us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:09:13.819491 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:09:13.819716 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:09:13.820408 (+ 692us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:09:13.820428 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:09:13.822031 (+ 1603us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:09:13.824181 (+ 2150us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:09:13.824190 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:09:13.824208 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:09:13.824586 (+ 378us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:09:13.824912 (+ 326us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:09:13.824916 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:09:13.824919 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:09:13.825093 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:09:13.825597 (+ 504us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:09:13.825603 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:09:13.825838 (+ 235us) client_negotiation.cc:769] Sending connection context
0620 06:09:13.826035 (+ 197us) client_negotiation.cc:240] Negotiation successful
0620 06:09:13.826332 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":791,"thread_start_us":132,"threads_started":1}
I20260620 06:09:13.828806 24130 heartbeater.cc:347] Connected to a master server at 127.11.102.62:40839
I20260620 06:09:13.829052 24130 heartbeater.cc:464] Registering TS with master...
I20260620 06:09:13.829641 24130 heartbeater.cc:511] Master 127.11.102.62:40839 requested a full tablet report, sending...
I20260620 06:09:13.831123 23916 ts_manager.cc:194] Registered new tserver with Master: c3a5167af887498198f016f81be7103f (127.11.102.1:33233)
I20260620 06:09:13.832283 23916 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.102.1@KRBTEST.COM'} at 127.11.102.1:54235
WARNING: no policy specified for kudu/127.11.102.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.102.2@KRBTEST.COM" created.
Jun 20 06:09:13 dist-test-slave-9hm5 krb5kdc[23440](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781935753, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.102.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.102.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935603120653-11672-0/krb5kdc/kudu_127.11.102.2.keytab.
Entry for principal kudu/127.11.102.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935603120653-11672-0/krb5kdc/kudu_127.11.102.2.keytab.
I20260620 06:09:13.859191 11672 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9HMp43/build/debug/bin/kudu
/tmp/dist-test-task9HMp43/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935603120653-11672-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935603120653-11672-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935603120653-11672-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935603120653-11672-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.11.102.20:53659/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.102.2:0
--local_ip_for_outbound_sockets=127.11.102.2
--webserver_interface=127.11.102.2
--webserver_port=0
--tserver_master_addrs=127.11.102.62:40839
--keytab_file=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935603120653-11672-0/krb5kdc/kudu_127.11.102.2.keytab
--principal=kudu/127.11.102.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.102.20:46459
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935603120653-11672-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935603120653-11672-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935603120653-11672-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 06:09:14.013672 24136 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:09:14.014005 24136 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:09:14.014071 24136 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:09:14.020087 24136 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:09:14.020154 24136 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:09:14.020236 24136 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.102.2
I20260620 06:09:14.027842 24136 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.102.20:46459
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935603120653-11672-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935603120653-11672-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.11.102.20:53659/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935603120653-11672-0/krb5kdc/kudu_127.11.102.2.keytab
--principal=kudu/127.11.102.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.102.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935603120653-11672-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.11.102.2
--webserver_port=0
--tserver_master_addrs=127.11.102.62:40839
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24136
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.102.2
--log_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935603120653-11672-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 05:28:48 UTC on ccycloud.quasar-fixhms.root.comops.site
build id 4457020
I20260620 06:09:14.029187 24136 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:09:14.030537 24136 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 06:09:14.042697 24142 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:09:14.042691 24143 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:09:14.043171 24145 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:09:14.043645 24136 server_base.cc:1034] running on GCE node
I20260620 06:09:14.044106 24136 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:09:14.044852 24136 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:09:14.051705 24136 hybrid_clock.cc:648] HybridClock initialized: now 1781935754051674 us; error 45 us; skew 500 ppm
Jun 20 06:09:14 dist-test-slave-9hm5 krb5kdc[23440](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935754, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.102.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 06:09:14.055634 24136 init.cc:377] Logged in from keytab as kudu/127.11.102.2@KRBTEST.COM (short username kudu)
I20260620 06:09:14.056926 24136 webserver.cc:460] Webserver started at http://127.11.102.2:40793/ using document root <none> and password file <none>
I20260620 06:09:14.057657 24136 fs_manager.cc:359] Metadata directory not provided
I20260620 06:09:14.057744 24136 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:09:14.058007 24136 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 06:09:14 dist-test-slave-9hm5 krb5kdc[23440](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935754, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.102.2@KRBTEST.COM for HTTP/127.11.102.20@KRBTEST.COM
I20260620 06:09:14.075975 24136 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935603120653-11672-0/minicluster-data/ts-1/data/instance:
uuid: "87154d0791e04b8586ac8a57691afadc"
format_stamp: "Formatted at 2026-06-20 06:09:14 on dist-test-slave-9hm5"
server_key: "9a61589d1f7eeb1c68fa1fcb225d18f6"
server_key_iv: "af2e6e9e85419163e257f1fd3facc5c9"
server_key_version: "kuduclusterkey@0"
I20260620 06:09:14.076756 24136 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935603120653-11672-0/minicluster-data/ts-1/wal/instance:
uuid: "87154d0791e04b8586ac8a57691afadc"
format_stamp: "Formatted at 2026-06-20 06:09:14 on dist-test-slave-9hm5"
server_key: "9a61589d1f7eeb1c68fa1fcb225d18f6"
server_key_iv: "af2e6e9e85419163e257f1fd3facc5c9"
server_key_version: "kuduclusterkey@0"
I20260620 06:09:14.082204 24136 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260620 06:09:14.098040 24152 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:09:14.099828 24136 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 06:09:14.100037 24136 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935603120653-11672-0/minicluster-data/ts-1/data,/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935603120653-11672-0/minicluster-data/ts-1/wal
uuid: "87154d0791e04b8586ac8a57691afadc"
format_stamp: "Formatted at 2026-06-20 06:09:14 on dist-test-slave-9hm5"
server_key: "9a61589d1f7eeb1c68fa1fcb225d18f6"
server_key_iv: "af2e6e9e85419163e257f1fd3facc5c9"
server_key_version: "kuduclusterkey@0"
I20260620 06:09:14.100250 24136 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935603120653-11672-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935603120653-11672-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935603120653-11672-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:09:14.142000 24136 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:09:14.146006 24136 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:09:14.146206 24136 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:09:14.146934 24136 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 06:09:14.148015 24136 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 06:09:14.148075 24136 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 06:09:14.148130 24136 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 06:09:14.148166 24136 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 06:09:14.159032 24136 rpc_server.cc:307] RPC server started. Bound to: 127.11.102.2:37915
I20260620 06:09:14.159229 24265 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.102.2:37915 every 8 connection(s)
I20260620 06:09:14.160260 24136 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935603120653-11672-0/minicluster-data/ts-1/data/info.pb
I20260620 06:09:14.169708 11672 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9HMp43/build/debug/bin/kudu as pid 24136
I20260620 06:09:14.169831 11672 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935603120653-11672-0/minicluster-data/ts-1/wal/instance
I20260620 06:09:14.170125 11672 external_mini_cluster.cc:1428] Setting key b04b72b73554c13642d035e1087732dc
Jun 20 06:09:14 dist-test-slave-9hm5 krb5kdc[23440](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935754, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.102.2@KRBTEST.COM for kudu/127.11.102.62@KRBTEST.COM
I20260620 06:09:14.182466 24132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:09:14.162303 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.102.2:46479 (local address 127.11.102.62:40839)
0620 06:09:14.162480 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0620 06:09:14.162486 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 06:09:14.163392 (+ 906us) server_negotiation.cc:407] Connection header received
0620 06:09:14.164599 (+ 1207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:09:14.164603 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:09:14.164662 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:09:14.164771 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:09:14.166241 (+ 1470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:09:14.167041 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:09:14.168851 (+ 1810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:09:14.169248 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:09:14.169342 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:09:14.176896 (+ 7554us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:09:14.176901 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:09:14.176904 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:09:14.176937 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:09:14.179660 (+ 2723us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:09:14.180493 (+ 833us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:09:14.180497 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:09:14.180499 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:09:14.180595 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:09:14.181060 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:09:14.181063 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:09:14.181065 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:09:14.181300 (+ 235us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:09:14.181383 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0620 06:09:14.182118 (+ 735us) server_negotiation.cc:299] Negotiation successful
0620 06:09:14.182265 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
WARNING: no policy specified for kudu/127.11.102.3@KRBTEST.COM; defaulting to no policy
I20260620 06:09:14.184221 24268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:09:14.162739 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.102.62:40839 (local address 127.11.102.2:46479)
0620 06:09:14.163223 (+ 484us) negotiation.cc:107] Waiting for socket to connect
0620 06:09:14.163288 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0620 06:09:14.164341 (+ 1053us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:09:14.164978 (+ 637us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:09:14.164987 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:09:14.165359 (+ 372us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:09:14.166035 (+ 676us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:09:14.166056 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:09:14.167198 (+ 1142us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:09:14.167202 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:09:14.168702 (+ 1500us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:09:14.168715 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:09:14.173820 (+ 5105us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:09:14.173824 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:09:14.174057 (+ 233us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:09:14.174811 (+ 754us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:09:14.174832 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:09:14.176651 (+ 1819us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:09:14.179961 (+ 3310us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:09:14.179969 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:09:14.179988 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:09:14.180362 (+ 374us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:09:14.180709 (+ 347us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:09:14.180712 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:09:14.180714 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:09:14.180938 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:09:14.181452 (+ 514us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:09:14.181459 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:09:14.181806 (+ 347us) client_negotiation.cc:769] Sending connection context
0620 06:09:14.182701 (+ 895us) client_negotiation.cc:240] Negotiation successful
0620 06:09:14.183072 (+ 371us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":273,"thread_start_us":136,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.102.3@KRBTEST.COM" created.
I20260620 06:09:14.185710 24266 heartbeater.cc:347] Connected to a master server at 127.11.102.62:40839
I20260620 06:09:14.185992 24266 heartbeater.cc:464] Registering TS with master...
I20260620 06:09:14.186575 24266 heartbeater.cc:511] Master 127.11.102.62:40839 requested a full tablet report, sending...
I20260620 06:09:14.188000 23916 ts_manager.cc:194] Registered new tserver with Master: 87154d0791e04b8586ac8a57691afadc (127.11.102.2:37915)
I20260620 06:09:14.188751 23916 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.102.2@KRBTEST.COM'} at 127.11.102.2:46479
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.102.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935603120653-11672-0/krb5kdc/kudu_127.11.102.3.keytab.
Entry for principal kudu/127.11.102.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935603120653-11672-0/krb5kdc/kudu_127.11.102.3.keytab.
I20260620 06:09:14.202903 11672 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9HMp43/build/debug/bin/kudu
/tmp/dist-test-task9HMp43/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935603120653-11672-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935603120653-11672-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935603120653-11672-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935603120653-11672-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.11.102.20:53659/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.102.3:0
--local_ip_for_outbound_sockets=127.11.102.3
--webserver_interface=127.11.102.3
--webserver_port=0
--tserver_master_addrs=127.11.102.62:40839
--keytab_file=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935603120653-11672-0/krb5kdc/kudu_127.11.102.3.keytab
--principal=kudu/127.11.102.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.102.20:46459
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935603120653-11672-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935603120653-11672-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935603120653-11672-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 06:09:14 dist-test-slave-9hm5 krb5kdc[23440](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935753, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.102.62@KRBTEST.COM for HTTP/127.11.102.20@KRBTEST.COM
W20260620 06:09:14.360898 24271 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:09:14.361258 24271 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:09:14.361325 24271 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:09:14.367270 24271 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:09:14.367352 24271 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:09:14.367436 24271 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.102.3
I20260620 06:09:14.375144 24271 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.102.20:46459
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935603120653-11672-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935603120653-11672-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.11.102.20:53659/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935603120653-11672-0/krb5kdc/kudu_127.11.102.3.keytab
--principal=kudu/127.11.102.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.102.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935603120653-11672-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.11.102.3
--webserver_port=0
--tserver_master_addrs=127.11.102.62:40839
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24271
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.102.3
--log_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935603120653-11672-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 05:28:48 UTC on ccycloud.quasar-fixhms.root.comops.site
build id 4457020
I20260620 06:09:14.376504 24271 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:09:14.377795 24271 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 06:09:14.390189 24277 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:09:14.390292 24276 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:09:14.390625 24279 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:09:14.390960 24271 server_base.cc:1034] running on GCE node
I20260620 06:09:14.391404 24271 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:09:14.392180 24271 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:09:14.393733 24271 hybrid_clock.cc:648] HybridClock initialized: now 1781935754393708 us; error 38 us; skew 500 ppm
Jun 20 06:09:14 dist-test-slave-9hm5 krb5kdc[23440](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935754, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.102.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 06:09:14.398197 24271 init.cc:377] Logged in from keytab as kudu/127.11.102.3@KRBTEST.COM (short username kudu)
I20260620 06:09:14.399228 24271 webserver.cc:460] Webserver started at http://127.11.102.3:38175/ using document root <none> and password file <none>
I20260620 06:09:14.399919 24271 fs_manager.cc:359] Metadata directory not provided
I20260620 06:09:14.399996 24271 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:09:14.400199 24271 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 06:09:14 dist-test-slave-9hm5 krb5kdc[23440](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935754, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.102.3@KRBTEST.COM for HTTP/127.11.102.20@KRBTEST.COM
I20260620 06:09:14.418035 24271 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935603120653-11672-0/minicluster-data/ts-2/data/instance:
uuid: "2198ba0c75d9450d81673132f3922db8"
format_stamp: "Formatted at 2026-06-20 06:09:14 on dist-test-slave-9hm5"
server_key: "89b355f2cc7b0d012e648b05364aac9e"
server_key_iv: "6db7726c835910ec2fd6ea702fb0f292"
server_key_version: "kuduclusterkey@0"
I20260620 06:09:14.418918 24271 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935603120653-11672-0/minicluster-data/ts-2/wal/instance:
uuid: "2198ba0c75d9450d81673132f3922db8"
format_stamp: "Formatted at 2026-06-20 06:09:14 on dist-test-slave-9hm5"
server_key: "89b355f2cc7b0d012e648b05364aac9e"
server_key_iv: "6db7726c835910ec2fd6ea702fb0f292"
server_key_version: "kuduclusterkey@0"
I20260620 06:09:14.424255 24271 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260620 06:09:14.443643 24286 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:09:14.445374 24271 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 06:09:14.445499 24271 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935603120653-11672-0/minicluster-data/ts-2/data,/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935603120653-11672-0/minicluster-data/ts-2/wal
uuid: "2198ba0c75d9450d81673132f3922db8"
format_stamp: "Formatted at 2026-06-20 06:09:14 on dist-test-slave-9hm5"
server_key: "89b355f2cc7b0d012e648b05364aac9e"
server_key_iv: "6db7726c835910ec2fd6ea702fb0f292"
server_key_version: "kuduclusterkey@0"
I20260620 06:09:14.445629 24271 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935603120653-11672-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935603120653-11672-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935603120653-11672-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:09:14.470822 24271 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:09:14.474578 24271 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:09:14.474776 24271 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:09:14.475453 24271 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 06:09:14.476480 24271 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 06:09:14.476548 24271 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 06:09:14.476603 24271 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 06:09:14.476641 24271 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 06:09:14.487649 24271 rpc_server.cc:307] RPC server started. Bound to: 127.11.102.3:44981
I20260620 06:09:14.488081 24399 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.102.3:44981 every 8 connection(s)
I20260620 06:09:14.488804 24271 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935603120653-11672-0/minicluster-data/ts-2/data/info.pb
I20260620 06:09:14.496531 11672 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9HMp43/build/debug/bin/kudu as pid 24271
I20260620 06:09:14.496651 11672 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935603120653-11672-0/minicluster-data/ts-2/wal/instance
I20260620 06:09:14.496964 11672 external_mini_cluster.cc:1428] Setting key a3997fd8e651272b044ea12f1c6086b4
Jun 20 06:09:14 dist-test-slave-9hm5 krb5kdc[23440](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935754, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.102.3@KRBTEST.COM for kudu/127.11.102.62@KRBTEST.COM
I20260620 06:09:14.504930 24132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:09:14.490796 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.102.3:37581 (local address 127.11.102.62:40839)
0620 06:09:14.490938 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0620 06:09:14.490943 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 06:09:14.491954 (+ 1011us) server_negotiation.cc:407] Connection header received
0620 06:09:14.493122 (+ 1168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:09:14.493125 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:09:14.493179 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:09:14.493285 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:09:14.494676 (+ 1391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:09:14.495490 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:09:14.497185 (+ 1695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:09:14.497529 (+ 344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:09:14.497610 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:09:14.500723 (+ 3113us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:09:14.500727 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:09:14.500730 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:09:14.500766 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:09:14.502527 (+ 1761us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:09:14.503146 (+ 619us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:09:14.503150 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:09:14.503152 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:09:14.503238 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:09:14.503656 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:09:14.503660 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:09:14.503662 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:09:14.503892 (+ 230us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:09:14.503974 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0620 06:09:14.504619 (+ 645us) server_negotiation.cc:299] Negotiation successful
0620 06:09:14.504776 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260620 06:09:14.506497 24402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:09:14.491252 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.102.62:40839 (local address 127.11.102.3:37581)
0620 06:09:14.491800 (+ 548us) negotiation.cc:107] Waiting for socket to connect
0620 06:09:14.491855 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0620 06:09:14.492881 (+ 1026us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:09:14.493492 (+ 611us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:09:14.493501 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:09:14.493837 (+ 336us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:09:14.494505 (+ 668us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:09:14.494523 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:09:14.495679 (+ 1156us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:09:14.495682 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:09:14.497045 (+ 1363us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:09:14.497056 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:09:14.497822 (+ 766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:09:14.497825 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:09:14.498021 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:09:14.498776 (+ 755us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:09:14.498796 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:09:14.500531 (+ 1735us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:09:14.502658 (+ 2127us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:09:14.502664 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:09:14.502680 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:09:14.503027 (+ 347us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:09:14.503346 (+ 319us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:09:14.503349 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:09:14.503351 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:09:14.503517 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:09:14.504039 (+ 522us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:09:14.504045 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:09:14.504341 (+ 296us) client_negotiation.cc:769] Sending connection context
0620 06:09:14.505142 (+ 801us) client_negotiation.cc:240] Negotiation successful
0620 06:09:14.505468 (+ 326us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":301,"thread_start_us":141,"threads_started":1}
I20260620 06:09:14.507802 24400 heartbeater.cc:347] Connected to a master server at 127.11.102.62:40839
I20260620 06:09:14.508038 24400 heartbeater.cc:464] Registering TS with master...
I20260620 06:09:14.508613 24400 heartbeater.cc:511] Master 127.11.102.62:40839 requested a full tablet report, sending...
I20260620 06:09:14.509646 23916 ts_manager.cc:194] Registered new tserver with Master: 2198ba0c75d9450d81673132f3922db8 (127.11.102.3:44981)
I20260620 06:09:14.510329 23916 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.102.3@KRBTEST.COM'} at 127.11.102.3:37581
I20260620 06:09:14.510866 11672 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 06:09:14.511642 11672 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9HMp43/build/debug/bin/kudu with pid 23997
I20260620 06:09:14.521941 11672 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9HMp43/build/debug/bin/kudu with pid 24136
I20260620 06:09:14.526857 11672 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9HMp43/build/debug/bin/kudu with pid 24271
I20260620 06:09:14.532843 11672 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9HMp43/build/debug/bin/kudu with pid 23884
I20260620 06:09:14.540738 11672 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9HMp43/build/debug/bin/kudu
/tmp/dist-test-task9HMp43/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935603120653-11672-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935603120653-11672-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935603120653-11672-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935603120653-11672-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.102.62:40839
--webserver_interface=127.11.102.62
--webserver_port=42523
--keytab_file=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935603120653-11672-0/krb5kdc/kudu_127.11.102.62.keytab
--principal=kudu/127.11.102.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.102.20:46459
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.102.62:40839
--ranger_config_path=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935603120653-11672-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.11.102.20:53659/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.11.102.20:53659/kms with env {KRB5CCNAME=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935603120653-11672-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935603120653-11672-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935603120653-11672-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 06:09:14.650419 24404 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:09:14.650687 24404 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:09:14.650727 24404 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:09:14.654702 24404 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 06:09:14.654774 24404 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:09:14.654800 24404 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:09:14.654816 24404 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 06:09:14.654831 24404 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 06:09:14.660358 24404 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.102.20:46459
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935603120653-11672-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935603120653-11672-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.11.102.20:53659/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.11.102.62:40839
--ranger_config_path=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935603120653-11672-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935603120653-11672-0/krb5kdc/kudu_127.11.102.62.keytab
--principal=kudu/127.11.102.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.102.62:40839
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935603120653-11672-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.102.62
--webserver_port=42523
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24404
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935603120653-11672-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 05:28:48 UTC on ccycloud.quasar-fixhms.root.comops.site
build id 4457020
I20260620 06:09:14.661379 24404 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:09:14.662333 24404 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 06:09:14.670732 24409 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:09:14.670738 24412 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:09:14.670746 24410 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:09:14.670794 24404 server_base.cc:1034] running on GCE node
I20260620 06:09:14.671538 24404 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:09:14.672672 24404 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:09:14.673862 24404 hybrid_clock.cc:648] HybridClock initialized: now 1781935754673843 us; error 31 us; skew 500 ppm
Jun 20 06:09:14 dist-test-slave-9hm5 krb5kdc[23440](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935754, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.102.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 06:09:14.676558 24404 init.cc:377] Logged in from keytab as kudu/127.11.102.62@KRBTEST.COM (short username kudu)
I20260620 06:09:14.677443 24404 webserver.cc:460] Webserver started at http://127.11.102.62:42523/ using document root <none> and password file <none>
I20260620 06:09:14.677973 24404 fs_manager.cc:359] Metadata directory not provided
I20260620 06:09:14.678064 24404 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 20 06:09:14 dist-test-slave-9hm5 krb5kdc[23440](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935754, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.102.62@KRBTEST.COM for HTTP/127.11.102.20@KRBTEST.COM
I20260620 06:09:14.756731 24404 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.001s
I20260620 06:09:14.759016 24420 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:09:14.760315 24404 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 06:09:14.760462 24404 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935603120653-11672-0/minicluster-data/master-0/data,/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935603120653-11672-0/minicluster-data/master-0/wal
uuid: "f42b7083570d4a95b735450329fb1c2f"
format_stamp: "Formatted at 2026-06-20 06:09:12 on dist-test-slave-9hm5"
server_key: "be182a1c82fc83eac5806eb34b42e7a1"
server_key_iv: "ffb27ff7200324dce2a19b4e79fde437"
server_key_version: "kuduclusterkey@0"
I20260620 06:09:14.761019 24404 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935603120653-11672-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935603120653-11672-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935603120653-11672-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:09:14.772329 24404 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:09:14.774871 24404 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:09:14.775070 24404 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:09:14.782251 24404 rpc_server.cc:307] RPC server started. Bound to: 127.11.102.62:40839
I20260620 06:09:14.782277 24472 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.102.62:40839 every 8 connection(s)
I20260620 06:09:14.783272 24404 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935603120653-11672-0/minicluster-data/master-0/data/info.pb
I20260620 06:09:14.787274 24473 sys_catalog.cc:263] Verifying existing consensus state
I20260620 06:09:14.788290 11672 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9HMp43/build/debug/bin/kudu as pid 24404
I20260620 06:09:14.788753 11672 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9HMp43/build/debug/bin/kudu
/tmp/dist-test-task9HMp43/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935603120653-11672-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935603120653-11672-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935603120653-11672-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935603120653-11672-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.11.102.20:53659/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.102.1:33233
--local_ip_for_outbound_sockets=127.11.102.1
--tserver_master_addrs=127.11.102.62:40839
--webserver_port=46761
--webserver_interface=127.11.102.1
--keytab_file=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935603120653-11672-0/krb5kdc/kudu_127.11.102.1.keytab
--principal=kudu/127.11.102.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.102.20:46459
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.11.102.20:53659/kms with env {KRB5CCNAME=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935603120653-11672-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935603120653-11672-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935603120653-11672-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 06:09:14.789034 24473 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f42b7083570d4a95b735450329fb1c2f: Bootstrap starting.
I20260620 06:09:14.805018 24473 log.cc:824] T 00000000000000000000000000000000 P f42b7083570d4a95b735450329fb1c2f: Log is configured to *not* fsync() on all Append() calls
I20260620 06:09:14.811511 24473 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f42b7083570d4a95b735450329fb1c2f: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 06:09:14.811940 24473 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f42b7083570d4a95b735450329fb1c2f: Bootstrap complete.
I20260620 06:09:14.816004 24473 raft_consensus.cc:348] T 00000000000000000000000000000000 P f42b7083570d4a95b735450329fb1c2f [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f42b7083570d4a95b735450329fb1c2f" member_type: VOTER last_known_addr { host: "127.11.102.62" port: 40839 } }
I20260620 06:09:14.816466 24473 raft_consensus.cc:729] T 00000000000000000000000000000000 P f42b7083570d4a95b735450329fb1c2f [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: f42b7083570d4a95b735450329fb1c2f, State: Initialized, Role: FOLLOWER
I20260620 06:09:14.816991 24473 consensus_queue.cc:260] T 00000000000000000000000000000000 P f42b7083570d4a95b735450329fb1c2f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f42b7083570d4a95b735450329fb1c2f" member_type: VOTER last_known_addr { host: "127.11.102.62" port: 40839 } }
I20260620 06:09:14.817158 24473 raft_consensus.cc:388] T 00000000000000000000000000000000 P f42b7083570d4a95b735450329fb1c2f [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 06:09:14.817257 24473 raft_consensus.cc:482] T 00000000000000000000000000000000 P f42b7083570d4a95b735450329fb1c2f [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 06:09:14.817378 24473 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f42b7083570d4a95b735450329fb1c2f [term 1 FOLLOWER]: Advancing to term 2
I20260620 06:09:14.819415 24473 raft_consensus.cc:504] T 00000000000000000000000000000000 P f42b7083570d4a95b735450329fb1c2f [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f42b7083570d4a95b735450329fb1c2f" member_type: VOTER last_known_addr { host: "127.11.102.62" port: 40839 } }
I20260620 06:09:14.819799 24473 leader_election.cc:302] T 00000000000000000000000000000000 P f42b7083570d4a95b735450329fb1c2f [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f42b7083570d4a95b735450329fb1c2f; no voters:
I20260620 06:09:14.820158 24473 leader_election.cc:288] T 00000000000000000000000000000000 P f42b7083570d4a95b735450329fb1c2f [CANDIDATE]: Term 2 election: Requested vote from peers
I20260620 06:09:14.820251 24477 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f42b7083570d4a95b735450329fb1c2f [term 2 FOLLOWER]: Leader election won for term 2
I20260620 06:09:14.820448 24477 raft_consensus.cc:686] T 00000000000000000000000000000000 P f42b7083570d4a95b735450329fb1c2f [term 2 LEADER]: Becoming Leader. State: Replica: f42b7083570d4a95b735450329fb1c2f, State: Running, Role: LEADER
I20260620 06:09:14.820801 24477 consensus_queue.cc:237] T 00000000000000000000000000000000 P f42b7083570d4a95b735450329fb1c2f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f42b7083570d4a95b735450329fb1c2f" member_type: VOTER last_known_addr { host: "127.11.102.62" port: 40839 } }
I20260620 06:09:14.821378 24473 sys_catalog.cc:564] T 00000000000000000000000000000000 P f42b7083570d4a95b735450329fb1c2f [sys.catalog]: configured and running, proceeding with master startup.
I20260620 06:09:14.822239 24479 sys_catalog.cc:455] T 00000000000000000000000000000000 P f42b7083570d4a95b735450329fb1c2f [sys.catalog]: SysCatalogTable state changed. Reason: New leader f42b7083570d4a95b735450329fb1c2f. Latest consensus state: current_term: 2 leader_uuid: "f42b7083570d4a95b735450329fb1c2f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f42b7083570d4a95b735450329fb1c2f" member_type: VOTER last_known_addr { host: "127.11.102.62" port: 40839 } } }
I20260620 06:09:14.822279 24478 sys_catalog.cc:455] T 00000000000000000000000000000000 P f42b7083570d4a95b735450329fb1c2f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "f42b7083570d4a95b735450329fb1c2f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f42b7083570d4a95b735450329fb1c2f" member_type: VOTER last_known_addr { host: "127.11.102.62" port: 40839 } } }
I20260620 06:09:14.822389 24479 sys_catalog.cc:458] T 00000000000000000000000000000000 P f42b7083570d4a95b735450329fb1c2f [sys.catalog]: This master's current role is: LEADER
I20260620 06:09:14.822398 24478 sys_catalog.cc:458] T 00000000000000000000000000000000 P f42b7083570d4a95b735450329fb1c2f [sys.catalog]: This master's current role is: LEADER
I20260620 06:09:14.822777 24486 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 06:09:14.825871 24486 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 06:09:14.826360 24473 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935603120653-11672-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260620 06:09:14.826535 24486 catalog_manager.cc:1259] Loaded cluster ID: b64f44e878a44f049260905de020040a
I20260620 06:09:14.826583 24486 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 06:09:14.832679 24486 catalog_manager.cc:1504] Loading token signing keys...
I20260620 06:09:14.833673 24486 catalog_manager.cc:5879] T 00000000000000000000000000000000 P f42b7083570d4a95b735450329fb1c2f: Loaded TSK: 0
I20260620 06:09:14.836865 24486 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260620 06:09:14.932224 24475 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:09:14.932614 24475 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:09:14.932699 24475 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:09:14.938970 24475 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:09:14.939059 24475 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:09:14.939162 24475 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.102.1
I20260620 06:09:14.947382 24475 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.102.20:46459
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935603120653-11672-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935603120653-11672-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.11.102.20:53659/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935603120653-11672-0/krb5kdc/kudu_127.11.102.1.keytab
--principal=kudu/127.11.102.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.102.1:33233
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935603120653-11672-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.11.102.1
--webserver_port=46761
--tserver_master_addrs=127.11.102.62:40839
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24475
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.102.1
--log_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935603120653-11672-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 05:28:48 UTC on ccycloud.quasar-fixhms.root.comops.site
build id 4457020
I20260620 06:09:14.948809 24475 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:09:14.950171 24475 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 06:09:14.963189 24475 server_base.cc:1034] running on GCE node
W20260620 06:09:14.963174 24513 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:09:14.963155 24511 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:09:14.968508 24510 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:09:14.969064 24475 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:09:14.969880 24475 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:09:14.971081 24475 hybrid_clock.cc:648] HybridClock initialized: now 1781935754971054 us; error 46 us; skew 500 ppm
Jun 20 06:09:14 dist-test-slave-9hm5 krb5kdc[23440](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935754, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.102.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 06:09:14.974639 24475 init.cc:377] Logged in from keytab as kudu/127.11.102.1@KRBTEST.COM (short username kudu)
I20260620 06:09:14.975850 24475 webserver.cc:460] Webserver started at http://127.11.102.1:46761/ using document root <none> and password file <none>
I20260620 06:09:14.976544 24475 fs_manager.cc:359] Metadata directory not provided
I20260620 06:09:14.976667 24475 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 20 06:09:14 dist-test-slave-9hm5 krb5kdc[23440](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935754, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.102.1@KRBTEST.COM for HTTP/127.11.102.20@KRBTEST.COM
I20260620 06:09:15.000777 24475 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.000s
I20260620 06:09:15.003732 24521 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:09:15.005108 24475 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 06:09:15.005259 24475 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935603120653-11672-0/minicluster-data/ts-0/data,/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935603120653-11672-0/minicluster-data/ts-0/wal
uuid: "c3a5167af887498198f016f81be7103f"
format_stamp: "Formatted at 2026-06-20 06:09:13 on dist-test-slave-9hm5"
server_key: "5f35d8f9e27a5f28f37b2db7dac0250e"
server_key_iv: "b30c368c808cea2ced7a871640e5eb46"
server_key_version: "kuduclusterkey@0"
I20260620 06:09:15.005894 24475 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935603120653-11672-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935603120653-11672-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935603120653-11672-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:09:15.027324 24475 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:09:15.031131 24475 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:09:15.031345 24475 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:09:15.032083 24475 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 06:09:15.033162 24475 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 06:09:15.033226 24475 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 06:09:15.033277 24475 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 06:09:15.033304 24475 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 06:09:15.043411 24475 rpc_server.cc:307] RPC server started. Bound to: 127.11.102.1:33233
I20260620 06:09:15.044775 24475 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935603120653-11672-0/minicluster-data/ts-0/data/info.pb
I20260620 06:09:15.045023 24634 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.102.1:33233 every 8 connection(s)
I20260620 06:09:15.050097 11672 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9HMp43/build/debug/bin/kudu as pid 24475
I20260620 06:09:15.050601 11672 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9HMp43/build/debug/bin/kudu
/tmp/dist-test-task9HMp43/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935603120653-11672-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935603120653-11672-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935603120653-11672-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935603120653-11672-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.11.102.20:53659/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.102.2:37915
--local_ip_for_outbound_sockets=127.11.102.2
--tserver_master_addrs=127.11.102.62:40839
--webserver_port=40793
--webserver_interface=127.11.102.2
--keytab_file=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935603120653-11672-0/krb5kdc/kudu_127.11.102.2.keytab
--principal=kudu/127.11.102.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.102.20:46459
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.11.102.20:53659/kms with env {KRB5CCNAME=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935603120653-11672-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935603120653-11672-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935603120653-11672-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 06:09:15 dist-test-slave-9hm5 krb5kdc[23440](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935754, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.102.1@KRBTEST.COM for kudu/127.11.102.62@KRBTEST.COM
I20260620 06:09:15.068244 24637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:09:15.048161 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.102.1:51661 (local address 127.11.102.62:40839)
0620 06:09:15.050467 (+ 2306us) server_negotiation.cc:206] Beginning negotiation
0620 06:09:15.050489 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0620 06:09:15.050530 (+ 41us) server_negotiation.cc:407] Connection header received
0620 06:09:15.050845 (+ 315us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:09:15.050860 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:09:15.051269 (+ 409us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:09:15.051635 (+ 366us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:09:15.053315 (+ 1680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:09:15.054882 (+ 1567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:09:15.056600 (+ 1718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:09:15.057383 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:09:15.057535 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:09:15.061901 (+ 4366us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:09:15.061907 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:09:15.061921 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:09:15.061948 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:09:15.063906 (+ 1958us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:09:15.064561 (+ 655us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:09:15.064568 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:09:15.064574 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:09:15.064742 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:09:15.065162 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:09:15.065166 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:09:15.065168 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:09:15.065647 (+ 479us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:09:15.065781 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0620 06:09:15.066633 (+ 852us) server_negotiation.cc:299] Negotiation successful
0620 06:09:15.066957 (+ 324us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2135,"thread_start_us":113,"threads_started":1}
I20260620 06:09:15.069751 24638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:09:15.048234 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.102.62:40839 (local address 127.11.102.1:51661)
0620 06:09:15.048847 (+ 613us) negotiation.cc:107] Waiting for socket to connect
0620 06:09:15.048903 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0620 06:09:15.049953 (+ 1050us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:09:15.052117 (+ 2164us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:09:15.052128 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:09:15.052477 (+ 349us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:09:15.053115 (+ 638us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:09:15.053136 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:09:15.055051 (+ 1915us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:09:15.055054 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:09:15.056451 (+ 1397us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:09:15.056464 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:09:15.057808 (+ 1344us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:09:15.057813 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:09:15.058017 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:09:15.059778 (+ 1761us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:09:15.059804 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:09:15.061664 (+ 1860us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:09:15.064048 (+ 2384us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:09:15.064055 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:09:15.064069 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:09:15.064415 (+ 346us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:09:15.064865 (+ 450us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:09:15.064868 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:09:15.064870 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:09:15.065043 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:09:15.065850 (+ 807us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:09:15.065857 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:09:15.066157 (+ 300us) client_negotiation.cc:769] Sending connection context
0620 06:09:15.068585 (+ 2428us) client_negotiation.cc:240] Negotiation successful
0620 06:09:15.068907 (+ 322us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":367,"thread_start_us":141,"threads_started":1}
I20260620 06:09:15.072643 24635 heartbeater.cc:347] Connected to a master server at 127.11.102.62:40839
I20260620 06:09:15.072927 24635 heartbeater.cc:464] Registering TS with master...
I20260620 06:09:15.073544 24635 heartbeater.cc:511] Master 127.11.102.62:40839 requested a full tablet report, sending...
W20260620 06:09:15.076020 24635 heartbeater.cc:644] Failed to heartbeat to 127.11.102.62:40839 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260620 06:09:15.076164 24635 heartbeater.cc:464] Registering TS with master...
I20260620 06:09:15.076373 24635 heartbeater.cc:511] Master 127.11.102.62:40839 requested a full tablet report, sending...
I20260620 06:09:15.077296 24635 heartbeater.cc:464] Registering TS with master...
I20260620 06:09:15.077464 24635 heartbeater.cc:511] Master 127.11.102.62:40839 requested a full tablet report, sending...
W20260620 06:09:15.078228 24635 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260620 06:09:15.228479 24639 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:09:15.228886 24639 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:09:15.228958 24639 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:09:15.233685 24639 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:09:15.233742 24639 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:09:15.233803 24639 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.102.2
I20260620 06:09:15.239519 24639 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.102.20:46459
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935603120653-11672-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935603120653-11672-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.11.102.20:53659/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935603120653-11672-0/krb5kdc/kudu_127.11.102.2.keytab
--principal=kudu/127.11.102.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.102.2:37915
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935603120653-11672-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.11.102.2
--webserver_port=40793
--tserver_master_addrs=127.11.102.62:40839
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24639
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.102.2
--log_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935603120653-11672-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 05:28:48 UTC on ccycloud.quasar-fixhms.root.comops.site
build id 4457020
I20260620 06:09:15.240703 24639 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:09:15.241984 24639 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 06:09:15.253326 24645 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:09:15.253328 24644 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:09:15.253890 24639 server_base.cc:1034] running on GCE node
W20260620 06:09:15.254026 24647 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:09:15.254544 24639 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:09:15.255335 24639 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:09:15.258682 24639 hybrid_clock.cc:648] HybridClock initialized: now 1781935755258585 us; error 91 us; skew 500 ppm
Jun 20 06:09:15 dist-test-slave-9hm5 krb5kdc[23440](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935755, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.102.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 06:09:15.262305 24639 init.cc:377] Logged in from keytab as kudu/127.11.102.2@KRBTEST.COM (short username kudu)
I20260620 06:09:15.263388 24639 webserver.cc:460] Webserver started at http://127.11.102.2:40793/ using document root <none> and password file <none>
I20260620 06:09:15.264139 24639 fs_manager.cc:359] Metadata directory not provided
I20260620 06:09:15.264259 24639 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 20 06:09:15 dist-test-slave-9hm5 krb5kdc[23440](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935755, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.102.2@KRBTEST.COM for HTTP/127.11.102.20@KRBTEST.COM
I20260620 06:09:15.289224 24639 fs_manager.cc:709] Time spent opening directory manager: real 0.008s user 0.002s sys 0.000s
I20260620 06:09:15.292332 24655 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:09:15.293890 24639 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 06:09:15.294018 24639 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935603120653-11672-0/minicluster-data/ts-1/data,/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935603120653-11672-0/minicluster-data/ts-1/wal
uuid: "87154d0791e04b8586ac8a57691afadc"
format_stamp: "Formatted at 2026-06-20 06:09:14 on dist-test-slave-9hm5"
server_key: "9a61589d1f7eeb1c68fa1fcb225d18f6"
server_key_iv: "af2e6e9e85419163e257f1fd3facc5c9"
server_key_version: "kuduclusterkey@0"
I20260620 06:09:15.294746 24639 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935603120653-11672-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935603120653-11672-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935603120653-11672-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:09:15.328783 24639 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:09:15.332087 24639 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:09:15.332309 24639 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:09:15.332943 24639 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 06:09:15.333937 24639 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 06:09:15.333997 24639 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 06:09:15.334052 24639 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 06:09:15.334079 24639 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 06:09:15.351749 24639 rpc_server.cc:307] RPC server started. Bound to: 127.11.102.2:37915
I20260620 06:09:15.352932 24639 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935603120653-11672-0/minicluster-data/ts-1/data/info.pb
I20260620 06:09:15.353837 24769 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.102.2:37915 every 8 connection(s)
I20260620 06:09:15.356994 11672 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9HMp43/build/debug/bin/kudu as pid 24639
I20260620 06:09:15.357510 11672 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9HMp43/build/debug/bin/kudu
/tmp/dist-test-task9HMp43/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935603120653-11672-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935603120653-11672-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935603120653-11672-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935603120653-11672-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.11.102.20:53659/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.102.3:44981
--local_ip_for_outbound_sockets=127.11.102.3
--tserver_master_addrs=127.11.102.62:40839
--webserver_port=38175
--webserver_interface=127.11.102.3
--keytab_file=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935603120653-11672-0/krb5kdc/kudu_127.11.102.3.keytab
--principal=kudu/127.11.102.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.102.20:46459
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.11.102.20:53659/kms with env {KRB5CCNAME=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935603120653-11672-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935603120653-11672-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935603120653-11672-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 06:09:15 dist-test-slave-9hm5 krb5kdc[23440](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935755, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.102.2@KRBTEST.COM for kudu/127.11.102.62@KRBTEST.COM
I20260620 06:09:15.377763 24637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:09:15.356546 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.102.2:54563 (local address 127.11.102.62:40839)
0620 06:09:15.356678 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0620 06:09:15.356683 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 06:09:15.356701 (+ 18us) server_negotiation.cc:407] Connection header received
0620 06:09:15.357507 (+ 806us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:09:15.357512 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:09:15.357574 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:09:15.357630 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:09:15.359895 (+ 2265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:09:15.360727 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:09:15.362392 (+ 1665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:09:15.362848 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:09:15.362986 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:09:15.366289 (+ 3303us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:09:15.366293 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:09:15.366296 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:09:15.366330 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:09:15.368240 (+ 1910us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:09:15.369001 (+ 761us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:09:15.369008 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:09:15.369011 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:09:15.369114 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:09:15.369564 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:09:15.369575 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:09:15.369580 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:09:15.369867 (+ 287us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:09:15.370010 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0620 06:09:15.377401 (+ 7391us) server_negotiation.cc:299] Negotiation successful
0620 06:09:15.377562 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260620 06:09:15.379853 24772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:09:15.355474 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.102.62:40839 (local address 127.11.102.2:54563)
0620 06:09:15.356016 (+ 542us) negotiation.cc:107] Waiting for socket to connect
0620 06:09:15.356070 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0620 06:09:15.357229 (+ 1159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:09:15.358340 (+ 1111us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:09:15.358353 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:09:15.359098 (+ 745us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:09:15.359694 (+ 596us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:09:15.359724 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:09:15.360902 (+ 1178us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:09:15.360905 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:09:15.362244 (+ 1339us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:09:15.362254 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:09:15.363196 (+ 942us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:09:15.363203 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:09:15.363457 (+ 254us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:09:15.364217 (+ 760us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:09:15.364244 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:09:15.366057 (+ 1813us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:09:15.368432 (+ 2375us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:09:15.368443 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:09:15.368462 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:09:15.368856 (+ 394us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:09:15.369225 (+ 369us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:09:15.369229 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:09:15.369231 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:09:15.369435 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:09:15.376586 (+ 7151us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:09:15.376595 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:09:15.377024 (+ 429us) client_negotiation.cc:769] Sending connection context
0620 06:09:15.378278 (+ 1254us) client_negotiation.cc:240] Negotiation successful
0620 06:09:15.378659 (+ 381us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":327,"thread_start_us":196,"threads_started":1}
I20260620 06:09:15.381261 24770 heartbeater.cc:347] Connected to a master server at 127.11.102.62:40839
I20260620 06:09:15.381517 24770 heartbeater.cc:464] Registering TS with master...
I20260620 06:09:15.382139 24770 heartbeater.cc:511] Master 127.11.102.62:40839 requested a full tablet report, sending...
W20260620 06:09:15.383949 24770 heartbeater.cc:644] Failed to heartbeat to 127.11.102.62:40839 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260620 06:09:15.384074 24770 heartbeater.cc:464] Registering TS with master...
I20260620 06:09:15.384244 24770 heartbeater.cc:511] Master 127.11.102.62:40839 requested a full tablet report, sending...
I20260620 06:09:15.385150 24770 heartbeater.cc:464] Registering TS with master...
I20260620 06:09:15.385322 24770 heartbeater.cc:511] Master 127.11.102.62:40839 requested a full tablet report, sending...
W20260620 06:09:15.386112 24770 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260620 06:09:15.495648 24773 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:09:15.496032 24773 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:09:15.496119 24773 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:09:15.502275 24773 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:09:15.502367 24773 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:09:15.502472 24773 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.102.3
I20260620 06:09:15.510545 24773 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.102.20:46459
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935603120653-11672-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935603120653-11672-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.11.102.20:53659/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935603120653-11672-0/krb5kdc/kudu_127.11.102.3.keytab
--principal=kudu/127.11.102.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.102.3:44981
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935603120653-11672-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.11.102.3
--webserver_port=38175
--tserver_master_addrs=127.11.102.62:40839
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24773
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.102.3
--log_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935603120653-11672-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 05:28:48 UTC on ccycloud.quasar-fixhms.root.comops.site
build id 4457020
I20260620 06:09:15.511811 24773 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:09:15.512899 24773 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 06:09:15.523123 24778 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:09:15.523674 24779 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:09:15.523818 24781 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:09:15.524091 24773 server_base.cc:1034] running on GCE node
I20260620 06:09:15.524597 24773 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:09:15.525388 24773 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:09:15.528702 24773 hybrid_clock.cc:648] HybridClock initialized: now 1781935755528672 us; error 39 us; skew 500 ppm
Jun 20 06:09:15 dist-test-slave-9hm5 krb5kdc[23440](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935755, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.102.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 06:09:15.531984 24773 init.cc:377] Logged in from keytab as kudu/127.11.102.3@KRBTEST.COM (short username kudu)
I20260620 06:09:15.532857 24773 webserver.cc:460] Webserver started at http://127.11.102.3:38175/ using document root <none> and password file <none>
I20260620 06:09:15.533437 24773 fs_manager.cc:359] Metadata directory not provided
I20260620 06:09:15.533551 24773 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 20 06:09:15 dist-test-slave-9hm5 krb5kdc[23440](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935755, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.102.3@KRBTEST.COM for HTTP/127.11.102.20@KRBTEST.COM
I20260620 06:09:15.555825 24773 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260620 06:09:15.564493 24789 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:09:15.566172 24773 fs_manager.cc:725] Time spent opening block manager: real 0.009s user 0.002s sys 0.000s
I20260620 06:09:15.566336 24773 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935603120653-11672-0/minicluster-data/ts-2/data,/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935603120653-11672-0/minicluster-data/ts-2/wal
uuid: "2198ba0c75d9450d81673132f3922db8"
format_stamp: "Formatted at 2026-06-20 06:09:14 on dist-test-slave-9hm5"
server_key: "89b355f2cc7b0d012e648b05364aac9e"
server_key_iv: "6db7726c835910ec2fd6ea702fb0f292"
server_key_version: "kuduclusterkey@0"
I20260620 06:09:15.567050 24773 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935603120653-11672-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935603120653-11672-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935603120653-11672-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:09:15.581513 24773 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:09:15.584619 24773 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:09:15.584828 24773 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:09:15.585445 24773 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 06:09:15.586284 24773 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 06:09:15.586356 24773 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 06:09:15.586439 24773 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 06:09:15.586483 24773 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 06:09:15.597886 24773 rpc_server.cc:307] RPC server started. Bound to: 127.11.102.3:44981
I20260620 06:09:15.598475 24902 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.102.3:44981 every 8 connection(s)
I20260620 06:09:15.599190 24773 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935603120653-11672-0/minicluster-data/ts-2/data/info.pb
I20260620 06:09:15.607688 11672 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9HMp43/build/debug/bin/kudu as pid 24773
Jun 20 06:09:15 dist-test-slave-9hm5 krb5kdc[23440](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935755, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.102.3@KRBTEST.COM for kudu/127.11.102.62@KRBTEST.COM
I20260620 06:09:15.615864 24637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:09:15.601353 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.102.3:41247 (local address 127.11.102.62:40839)
0620 06:09:15.601492 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0620 06:09:15.601498 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 06:09:15.602592 (+ 1094us) server_negotiation.cc:407] Connection header received
0620 06:09:15.603941 (+ 1349us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:09:15.603945 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:09:15.604005 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:09:15.604061 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:09:15.605750 (+ 1689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:09:15.606549 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:09:15.608887 (+ 2338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:09:15.609115 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:09:15.609263 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:09:15.611942 (+ 2679us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:09:15.611946 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:09:15.611949 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:09:15.611977 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:09:15.613534 (+ 1557us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:09:15.614147 (+ 613us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:09:15.614151 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:09:15.614153 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:09:15.614241 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:09:15.614646 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:09:15.614649 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:09:15.614651 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:09:15.614846 (+ 195us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:09:15.614934 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0620 06:09:15.615551 (+ 617us) server_negotiation.cc:299] Negotiation successful
0620 06:09:15.615701 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260620 06:09:15.616891 24905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:09:15.601880 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.102.62:40839 (local address 127.11.102.3:41247)
0620 06:09:15.602416 (+ 536us) negotiation.cc:107] Waiting for socket to connect
0620 06:09:15.602475 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0620 06:09:15.603651 (+ 1176us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:09:15.604483 (+ 832us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:09:15.604493 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:09:15.604854 (+ 361us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:09:15.605569 (+ 715us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:09:15.605591 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:09:15.606716 (+ 1125us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:09:15.606720 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:09:15.608759 (+ 2039us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:09:15.608769 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:09:15.609280 (+ 511us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:09:15.609284 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:09:15.609548 (+ 264us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:09:15.610162 (+ 614us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:09:15.610182 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:09:15.611770 (+ 1588us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:09:15.613652 (+ 1882us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:09:15.613660 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:09:15.613677 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:09:15.614017 (+ 340us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:09:15.614347 (+ 330us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:09:15.614351 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:09:15.614353 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:09:15.614527 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:09:15.614954 (+ 427us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:09:15.614961 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:09:15.615268 (+ 307us) client_negotiation.cc:769] Sending connection context
0620 06:09:15.615465 (+ 197us) client_negotiation.cc:240] Negotiation successful
0620 06:09:15.615811 (+ 346us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":306,"thread_start_us":151,"threads_started":1}
I20260620 06:09:15.618193 24903 heartbeater.cc:347] Connected to a master server at 127.11.102.62:40839
I20260620 06:09:15.618424 24903 heartbeater.cc:464] Registering TS with master...
I20260620 06:09:15.619041 24903 heartbeater.cc:511] Master 127.11.102.62:40839 requested a full tablet report, sending...
I20260620 06:09:15.620594 24437 ts_manager.cc:194] Registered new tserver with Master: 2198ba0c75d9450d81673132f3922db8 (127.11.102.3:44981)
I20260620 06:09:15.622144 24914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:09:15.608401 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55210 (local address 127.11.102.62:40839)
0620 06:09:15.612751 (+ 4350us) server_negotiation.cc:206] Beginning negotiation
0620 06:09:15.612762 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 06:09:15.612776 (+ 14us) server_negotiation.cc:407] Connection header received
0620 06:09:15.612814 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:09:15.612820 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:09:15.612874 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:09:15.612969 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:09:15.614345 (+ 1376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:09:15.615030 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:09:15.616289 (+ 1259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:09:15.616520 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:09:15.616625 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:09:15.618149 (+ 1524us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:09:15.618155 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:09:15.618162 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:09:15.618187 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:09:15.619731 (+ 1544us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:09:15.620239 (+ 508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:09:15.620243 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:09:15.620245 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:09:15.620337 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:09:15.621231 (+ 894us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:09:15.621235 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:09:15.621237 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:09:15.621479 (+ 242us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:09:15.621655 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0620 06:09:15.621850 (+ 195us) server_negotiation.cc:299] Negotiation successful
0620 06:09:15.621968 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4257,"thread_start_us":94,"threads_started":1}
I20260620 06:09:15.622114 24437 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.102.3@KRBTEST.COM'} at 127.11.102.3:41247
Jun 20 06:09:15 dist-test-slave-9hm5 krb5kdc[23440](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781935755, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.102.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 06:09:16.078472 24635 heartbeater.cc:464] Registering TS with master...
I20260620 06:09:16.078794 24635 heartbeater.cc:511] Master 127.11.102.62:40839 requested a full tablet report, sending...
I20260620 06:09:16.079538 24437 ts_manager.cc:194] Registered new tserver with Master: c3a5167af887498198f016f81be7103f (127.11.102.1:33233)
I20260620 06:09:16.080399 24437 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.102.1@KRBTEST.COM'} at 127.11.102.1:51661
Jun 20 06:09:16 dist-test-slave-9hm5 krb5kdc[23440](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935755, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.102.62@KRBTEST.COM for HTTP/127.11.102.20@KRBTEST.COM
I20260620 06:09:16.386333 24770 heartbeater.cc:464] Registering TS with master...
I20260620 06:09:16.386629 24770 heartbeater.cc:511] Master 127.11.102.62:40839 requested a full tablet report, sending...
I20260620 06:09:16.387362 24437 ts_manager.cc:194] Registered new tserver with Master: 87154d0791e04b8586ac8a57691afadc (127.11.102.2:37915)
I20260620 06:09:16.388021 24437 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.102.2@KRBTEST.COM'} at 127.11.102.2:54563
I20260620 06:09:16.396845 11672 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 06:09:16.407977 24925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:09:16.398552 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55216 (local address 127.11.102.62:40839)
0620 06:09:16.399113 (+ 561us) server_negotiation.cc:206] Beginning negotiation
0620 06:09:16.399118 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 06:09:16.399129 (+ 11us) server_negotiation.cc:407] Connection header received
0620 06:09:16.399164 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:09:16.399167 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:09:16.399222 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:09:16.399296 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:09:16.400208 (+ 912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:09:16.400974 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:09:16.402149 (+ 1175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:09:16.402487 (+ 338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:09:16.402575 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:09:16.404358 (+ 1783us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:09:16.404361 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:09:16.404365 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:09:16.404391 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:09:16.406145 (+ 1754us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:09:16.406654 (+ 509us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:09:16.406658 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:09:16.406660 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:09:16.406751 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:09:16.407055 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:09:16.407058 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:09:16.407060 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:09:16.407260 (+ 200us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:09:16.407379 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0620 06:09:16.407678 (+ 299us) server_negotiation.cc:299] Negotiation successful
0620 06:09:16.407814 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":493,"thread_start_us":72,"threads_started":1}
I20260620 06:09:16.412200 24437 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55216:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 06:09:16.417071 24437 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 06:09:16.437860 24930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:09:16.428077 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.102.1:33233 (local address 127.0.0.1:36048)
0620 06:09:16.431802 (+ 3725us) negotiation.cc:107] Waiting for socket to connect
0620 06:09:16.431816 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 06:09:16.431906 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:09:16.432581 (+ 675us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:09:16.432586 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:09:16.432607 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 06:09:16.432748 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:09:16.432754 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:09:16.434117 (+ 1363us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:09:16.434120 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:09:16.435456 (+ 1336us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:09:16.435467 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:09:16.437301 (+ 1834us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:09:16.437304 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:09:16.437424 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:09:16.437463 (+ 39us) client_negotiation.cc:769] Sending connection context
0620 06:09:16.437569 (+ 106us) client_negotiation.cc:240] Negotiation successful
0620 06:09:16.437686 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3663,"thread_start_us":95,"threads_started":1}
I20260620 06:09:16.439355 24934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:09:16.428068 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.102.2:37915 (local address 127.0.0.1:58452)
0620 06:09:16.429888 (+ 1820us) negotiation.cc:107] Waiting for socket to connect
0620 06:09:16.429939 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0620 06:09:16.430063 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:09:16.433362 (+ 3299us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:09:16.433366 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:09:16.433384 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 06:09:16.433474 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:09:16.433480 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:09:16.436016 (+ 2536us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:09:16.436019 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:09:16.437184 (+ 1165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:09:16.437195 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:09:16.438970 (+ 1775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:09:16.438974 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:09:16.439068 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:09:16.439092 (+ 24us) client_negotiation.cc:769] Sending connection context
0620 06:09:16.439156 (+ 64us) client_negotiation.cc:240] Negotiation successful
0620 06:09:16.439222 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1618,"thread_start_us":121,"threads_started":1}
I20260620 06:09:16.440181 24932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:09:16.428872 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58452 (local address 127.11.102.2:37915)
0620 06:09:16.432717 (+ 3845us) server_negotiation.cc:206] Beginning negotiation
0620 06:09:16.432724 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 06:09:16.432787 (+ 63us) server_negotiation.cc:407] Connection header received
0620 06:09:16.432864 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:09:16.432869 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:09:16.433050 (+ 181us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:09:16.433192 (+ 142us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 06:09:16.433621 (+ 429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:09:16.434798 (+ 1177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:09:16.437334 (+ 2536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:09:16.438449 (+ 1115us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:09:16.438553 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:09:16.439878 (+ 1325us) server_negotiation.cc:1035] Waiting for connection context
0620 06:09:16.439939 (+ 61us) server_negotiation.cc:299] Negotiation successful
0620 06:09:16.440040 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3547,"thread_start_us":67,"threads_started":1}
I20260620 06:09:16.440810 24933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:09:16.428863 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36048 (local address 127.11.102.1:33233)
0620 06:09:16.430514 (+ 1651us) server_negotiation.cc:206] Beginning negotiation
0620 06:09:16.430521 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 06:09:16.432003 (+ 1482us) server_negotiation.cc:407] Connection header received
0620 06:09:16.432083 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:09:16.432088 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:09:16.432301 (+ 213us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:09:16.432430 (+ 129us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 06:09:16.432866 (+ 436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:09:16.433987 (+ 1121us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:09:16.435592 (+ 1605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:09:16.436434 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:09:16.436537 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:09:16.436680 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0620 06:09:16.440556 (+ 3876us) server_negotiation.cc:299] Negotiation successful
0620 06:09:16.440660 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1563,"thread_start_us":82,"threads_started":1}
I20260620 06:09:16.442902 24931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:09:16.429503 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.102.3:44981 (local address 127.0.0.1:43848)
0620 06:09:16.435706 (+ 6203us) negotiation.cc:107] Waiting for socket to connect
0620 06:09:16.435721 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 06:09:16.435837 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:09:16.438163 (+ 2326us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:09:16.438166 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:09:16.438184 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 06:09:16.438266 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:09:16.438272 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:09:16.440280 (+ 2008us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:09:16.440283 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:09:16.441337 (+ 1054us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:09:16.441348 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:09:16.442519 (+ 1171us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:09:16.442522 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:09:16.442617 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:09:16.442639 (+ 22us) client_negotiation.cc:769] Sending connection context
0620 06:09:16.442720 (+ 81us) client_negotiation.cc:240] Negotiation successful
0620 06:09:16.442782 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6140,"thread_start_us":83,"threads_started":1}
I20260620 06:09:16.443552 24704 tablet_service.cc:1467] Processing CreateTablet for tablet db32a415010a47e995de7ca8d2bbf1ed (DEFAULT_TABLE table=test-table [id=c5cbb8ee9d6c4d46ae8a4234ba353425]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 06:09:16.444074 24935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:09:16.429870 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43848 (local address 127.11.102.3:44981)
0620 06:09:16.430350 (+ 480us) server_negotiation.cc:206] Beginning negotiation
0620 06:09:16.430357 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 06:09:16.436748 (+ 6391us) server_negotiation.cc:407] Connection header received
0620 06:09:16.436831 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:09:16.436837 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:09:16.437017 (+ 180us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:09:16.437151 (+ 134us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 06:09:16.438404 (+ 1253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:09:16.440135 (+ 1731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:09:16.441482 (+ 1347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:09:16.442234 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:09:16.442334 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:09:16.442457 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0620 06:09:16.443115 (+ 658us) server_negotiation.cc:299] Negotiation successful
0620 06:09:16.443208 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":390,"thread_start_us":76,"threads_started":1}
I20260620 06:09:16.443761 24567 tablet_service.cc:1467] Processing CreateTablet for tablet db32a415010a47e995de7ca8d2bbf1ed (DEFAULT_TABLE table=test-table [id=c5cbb8ee9d6c4d46ae8a4234ba353425]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 06:09:16.444777 24567 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet db32a415010a47e995de7ca8d2bbf1ed. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 06:09:16.444778 24704 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet db32a415010a47e995de7ca8d2bbf1ed. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 06:09:16.446501 24819 tablet_service.cc:1467] Processing CreateTablet for tablet db32a415010a47e995de7ca8d2bbf1ed (DEFAULT_TABLE table=test-table [id=c5cbb8ee9d6c4d46ae8a4234ba353425]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 06:09:16.447422 24819 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet db32a415010a47e995de7ca8d2bbf1ed. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 06:09:16.456382 24936 tablet_bootstrap.cc:492] T db32a415010a47e995de7ca8d2bbf1ed P 87154d0791e04b8586ac8a57691afadc: Bootstrap starting.
I20260620 06:09:16.458559 24938 tablet_bootstrap.cc:492] T db32a415010a47e995de7ca8d2bbf1ed P 2198ba0c75d9450d81673132f3922db8: Bootstrap starting.
I20260620 06:09:16.458789 24936 tablet_bootstrap.cc:654] T db32a415010a47e995de7ca8d2bbf1ed P 87154d0791e04b8586ac8a57691afadc: Neither blocks nor log segments found. Creating new log.
I20260620 06:09:16.459738 24936 log.cc:824] T db32a415010a47e995de7ca8d2bbf1ed P 87154d0791e04b8586ac8a57691afadc: Log is configured to *not* fsync() on all Append() calls
I20260620 06:09:16.460647 24938 tablet_bootstrap.cc:654] T db32a415010a47e995de7ca8d2bbf1ed P 2198ba0c75d9450d81673132f3922db8: Neither blocks nor log segments found. Creating new log.
I20260620 06:09:16.461411 24938 log.cc:824] T db32a415010a47e995de7ca8d2bbf1ed P 2198ba0c75d9450d81673132f3922db8: Log is configured to *not* fsync() on all Append() calls
I20260620 06:09:16.462415 24936 tablet_bootstrap.cc:492] T db32a415010a47e995de7ca8d2bbf1ed P 87154d0791e04b8586ac8a57691afadc: No bootstrap required, opened a new log
I20260620 06:09:16.462623 24936 ts_tablet_manager.cc:1397] T db32a415010a47e995de7ca8d2bbf1ed P 87154d0791e04b8586ac8a57691afadc: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260620 06:09:16.463228 24938 tablet_bootstrap.cc:492] T db32a415010a47e995de7ca8d2bbf1ed P 2198ba0c75d9450d81673132f3922db8: No bootstrap required, opened a new log
I20260620 06:09:16.463400 24938 ts_tablet_manager.cc:1397] T db32a415010a47e995de7ca8d2bbf1ed P 2198ba0c75d9450d81673132f3922db8: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260620 06:09:16.463963 24937 tablet_bootstrap.cc:492] T db32a415010a47e995de7ca8d2bbf1ed P c3a5167af887498198f016f81be7103f: Bootstrap starting.
I20260620 06:09:16.466140 24937 tablet_bootstrap.cc:654] T db32a415010a47e995de7ca8d2bbf1ed P c3a5167af887498198f016f81be7103f: Neither blocks nor log segments found. Creating new log.
I20260620 06:09:16.466931 24937 log.cc:824] T db32a415010a47e995de7ca8d2bbf1ed P c3a5167af887498198f016f81be7103f: Log is configured to *not* fsync() on all Append() calls
I20260620 06:09:16.466807 24936 raft_consensus.cc:348] T db32a415010a47e995de7ca8d2bbf1ed P 87154d0791e04b8586ac8a57691afadc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87154d0791e04b8586ac8a57691afadc" member_type: VOTER last_known_addr { host: "127.11.102.2" port: 37915 } } peers { permanent_uuid: "c3a5167af887498198f016f81be7103f" member_type: VOTER last_known_addr { host: "127.11.102.1" port: 33233 } } peers { permanent_uuid: "2198ba0c75d9450d81673132f3922db8" member_type: VOTER last_known_addr { host: "127.11.102.3" port: 44981 } }
I20260620 06:09:16.467067 24936 raft_consensus.cc:374] T db32a415010a47e995de7ca8d2bbf1ed P 87154d0791e04b8586ac8a57691afadc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 06:09:16.467118 24936 raft_consensus.cc:729] T db32a415010a47e995de7ca8d2bbf1ed P 87154d0791e04b8586ac8a57691afadc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 87154d0791e04b8586ac8a57691afadc, State: Initialized, Role: FOLLOWER
I20260620 06:09:16.467651 24936 consensus_queue.cc:260] T db32a415010a47e995de7ca8d2bbf1ed P 87154d0791e04b8586ac8a57691afadc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87154d0791e04b8586ac8a57691afadc" member_type: VOTER last_known_addr { host: "127.11.102.2" port: 37915 } } peers { permanent_uuid: "c3a5167af887498198f016f81be7103f" member_type: VOTER last_known_addr { host: "127.11.102.1" port: 33233 } } peers { permanent_uuid: "2198ba0c75d9450d81673132f3922db8" member_type: VOTER last_known_addr { host: "127.11.102.3" port: 44981 } }
I20260620 06:09:16.468459 24938 raft_consensus.cc:348] T db32a415010a47e995de7ca8d2bbf1ed P 2198ba0c75d9450d81673132f3922db8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87154d0791e04b8586ac8a57691afadc" member_type: VOTER last_known_addr { host: "127.11.102.2" port: 37915 } } peers { permanent_uuid: "c3a5167af887498198f016f81be7103f" member_type: VOTER last_known_addr { host: "127.11.102.1" port: 33233 } } peers { permanent_uuid: "2198ba0c75d9450d81673132f3922db8" member_type: VOTER last_known_addr { host: "127.11.102.3" port: 44981 } }
I20260620 06:09:16.468694 24936 ts_tablet_manager.cc:1428] T db32a415010a47e995de7ca8d2bbf1ed P 87154d0791e04b8586ac8a57691afadc: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260620 06:09:16.468706 24938 raft_consensus.cc:374] T db32a415010a47e995de7ca8d2bbf1ed P 2198ba0c75d9450d81673132f3922db8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 06:09:16.468760 24938 raft_consensus.cc:729] T db32a415010a47e995de7ca8d2bbf1ed P 2198ba0c75d9450d81673132f3922db8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2198ba0c75d9450d81673132f3922db8, State: Initialized, Role: FOLLOWER
I20260620 06:09:16.469050 24770 heartbeater.cc:503] Master 127.11.102.62:40839 was elected leader, sending a full tablet report...
I20260620 06:09:16.469178 24938 consensus_queue.cc:260] T db32a415010a47e995de7ca8d2bbf1ed P 2198ba0c75d9450d81673132f3922db8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87154d0791e04b8586ac8a57691afadc" member_type: VOTER last_known_addr { host: "127.11.102.2" port: 37915 } } peers { permanent_uuid: "c3a5167af887498198f016f81be7103f" member_type: VOTER last_known_addr { host: "127.11.102.1" port: 33233 } } peers { permanent_uuid: "2198ba0c75d9450d81673132f3922db8" member_type: VOTER last_known_addr { host: "127.11.102.3" port: 44981 } }
I20260620 06:09:16.470304 24938 ts_tablet_manager.cc:1428] T db32a415010a47e995de7ca8d2bbf1ed P 2198ba0c75d9450d81673132f3922db8: Time spent starting tablet: real 0.007s user 0.003s sys 0.003s
I20260620 06:09:16.470619 24903 heartbeater.cc:503] Master 127.11.102.62:40839 was elected leader, sending a full tablet report...
I20260620 06:09:16.475093 24937 tablet_bootstrap.cc:492] T db32a415010a47e995de7ca8d2bbf1ed P c3a5167af887498198f016f81be7103f: No bootstrap required, opened a new log
I20260620 06:09:16.475323 24937 ts_tablet_manager.cc:1397] T db32a415010a47e995de7ca8d2bbf1ed P c3a5167af887498198f016f81be7103f: Time spent bootstrapping tablet: real 0.012s user 0.005s sys 0.000s
I20260620 06:09:16.478432 24937 raft_consensus.cc:348] T db32a415010a47e995de7ca8d2bbf1ed P c3a5167af887498198f016f81be7103f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87154d0791e04b8586ac8a57691afadc" member_type: VOTER last_known_addr { host: "127.11.102.2" port: 37915 } } peers { permanent_uuid: "c3a5167af887498198f016f81be7103f" member_type: VOTER last_known_addr { host: "127.11.102.1" port: 33233 } } peers { permanent_uuid: "2198ba0c75d9450d81673132f3922db8" member_type: VOTER last_known_addr { host: "127.11.102.3" port: 44981 } }
I20260620 06:09:16.478667 24937 raft_consensus.cc:374] T db32a415010a47e995de7ca8d2bbf1ed P c3a5167af887498198f016f81be7103f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 06:09:16.478719 24937 raft_consensus.cc:729] T db32a415010a47e995de7ca8d2bbf1ed P c3a5167af887498198f016f81be7103f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c3a5167af887498198f016f81be7103f, State: Initialized, Role: FOLLOWER
I20260620 06:09:16.479146 24937 consensus_queue.cc:260] T db32a415010a47e995de7ca8d2bbf1ed P c3a5167af887498198f016f81be7103f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87154d0791e04b8586ac8a57691afadc" member_type: VOTER last_known_addr { host: "127.11.102.2" port: 37915 } } peers { permanent_uuid: "c3a5167af887498198f016f81be7103f" member_type: VOTER last_known_addr { host: "127.11.102.1" port: 33233 } } peers { permanent_uuid: "2198ba0c75d9450d81673132f3922db8" member_type: VOTER last_known_addr { host: "127.11.102.3" port: 44981 } }
I20260620 06:09:16.479831 24635 heartbeater.cc:503] Master 127.11.102.62:40839 was elected leader, sending a full tablet report...
I20260620 06:09:16.480139 24937 ts_tablet_manager.cc:1428] T db32a415010a47e995de7ca8d2bbf1ed P c3a5167af887498198f016f81be7103f: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
W20260620 06:09:16.547168 24636 tablet.cc:2369] T db32a415010a47e995de7ca8d2bbf1ed P c3a5167af887498198f016f81be7103f: Can't schedule compaction. Clean time has not been advanced past its initial value.
Jun 20 06:09:16 dist-test-slave-9hm5 krb5kdc[23440](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935755, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.102.62@KRBTEST.COM for HTTP/127.11.102.20@KRBTEST.COM
W20260620 06:09:16.600713 24904 tablet.cc:2369] T db32a415010a47e995de7ca8d2bbf1ed P 2198ba0c75d9450d81673132f3922db8: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 06:09:16.604689 24771 tablet.cc:2369] T db32a415010a47e995de7ca8d2bbf1ed P 87154d0791e04b8586ac8a57691afadc: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 06:09:16.629880 24943 raft_consensus.cc:482] T db32a415010a47e995de7ca8d2bbf1ed P 2198ba0c75d9450d81673132f3922db8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 06:09:16.630142 24943 raft_consensus.cc:504] T db32a415010a47e995de7ca8d2bbf1ed P 2198ba0c75d9450d81673132f3922db8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87154d0791e04b8586ac8a57691afadc" member_type: VOTER last_known_addr { host: "127.11.102.2" port: 37915 } } peers { permanent_uuid: "c3a5167af887498198f016f81be7103f" member_type: VOTER last_known_addr { host: "127.11.102.1" port: 33233 } } peers { permanent_uuid: "2198ba0c75d9450d81673132f3922db8" member_type: VOTER last_known_addr { host: "127.11.102.3" port: 44981 } }
I20260620 06:09:16.631620 24943 leader_election.cc:288] T db32a415010a47e995de7ca8d2bbf1ed P 2198ba0c75d9450d81673132f3922db8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 87154d0791e04b8586ac8a57691afadc (127.11.102.2:37915), c3a5167af887498198f016f81be7103f (127.11.102.1:33233)
I20260620 06:09:16.646953 24933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:09:16.632581 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.102.3:47033 (local address 127.11.102.1:33233)
0620 06:09:16.632707 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0620 06:09:16.632711 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 06:09:16.640009 (+ 7298us) server_negotiation.cc:407] Connection header received
0620 06:09:16.640217 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:09:16.640221 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:09:16.640273 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:09:16.640320 (+ 47us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 06:09:16.643369 (+ 3049us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:09:16.644126 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:09:16.645625 (+ 1499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:09:16.646212 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:09:16.646315 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:09:16.646377 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0620 06:09:16.646714 (+ 337us) server_negotiation.cc:299] Negotiation successful
0620 06:09:16.646813 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260620 06:09:16.647042 24932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:09:16.632244 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.102.3:59709 (local address 127.11.102.2:37915)
0620 06:09:16.632362 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0620 06:09:16.632366 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 06:09:16.632598 (+ 232us) server_negotiation.cc:407] Connection header received
0620 06:09:16.633100 (+ 502us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:09:16.633104 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:09:16.633155 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:09:16.633205 (+ 50us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 06:09:16.633594 (+ 389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:09:16.634400 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:09:16.645693 (+ 11293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:09:16.646338 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:09:16.646443 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:09:16.646491 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0620 06:09:16.646835 (+ 344us) server_negotiation.cc:299] Negotiation successful
0620 06:09:16.646915 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260620 06:09:16.647480 24947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:09:16.631822 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.102.2:37915 (local address 127.11.102.3:59709)
0620 06:09:16.632503 (+ 681us) negotiation.cc:107] Waiting for socket to connect
0620 06:09:16.632517 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 06:09:16.632783 (+ 266us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:09:16.633346 (+ 563us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:09:16.633348 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:09:16.633367 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 06:09:16.633461 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:09:16.633467 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:09:16.643597 (+ 10130us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:09:16.643601 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:09:16.645047 (+ 1446us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:09:16.645057 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:09:16.646599 (+ 1542us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:09:16.646603 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:09:16.646693 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:09:16.646715 (+ 22us) client_negotiation.cc:769] Sending connection context
0620 06:09:16.647244 (+ 529us) client_negotiation.cc:240] Negotiation successful
0620 06:09:16.647324 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":616,"mutex_wait_us":4,"thread_start_us":124,"threads_started":1}
I20260620 06:09:16.647985 24948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:09:16.632054 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.102.1:33233 (local address 127.11.102.3:47033)
0620 06:09:16.639910 (+ 7856us) negotiation.cc:107] Waiting for socket to connect
0620 06:09:16.639927 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 06:09:16.640093 (+ 166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:09:16.640457 (+ 364us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:09:16.640460 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:09:16.640475 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 06:09:16.643209 (+ 2734us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:09:16.643218 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:09:16.644278 (+ 1060us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:09:16.644281 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:09:16.645482 (+ 1201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:09:16.645495 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:09:16.646479 (+ 984us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:09:16.646482 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:09:16.646570 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:09:16.646592 (+ 22us) client_negotiation.cc:769] Sending connection context
0620 06:09:16.647478 (+ 886us) client_negotiation.cc:240] Negotiation successful
0620 06:09:16.647840 (+ 362us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7763,"thread_start_us":113,"threads_started":1}
I20260620 06:09:16.648172 24718 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db32a415010a47e995de7ca8d2bbf1ed" candidate_uuid: "2198ba0c75d9450d81673132f3922db8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87154d0791e04b8586ac8a57691afadc" is_pre_election: true
I20260620 06:09:16.648511 24718 raft_consensus.cc:2454] T db32a415010a47e995de7ca8d2bbf1ed P 87154d0791e04b8586ac8a57691afadc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2198ba0c75d9450d81673132f3922db8 in term 0.
I20260620 06:09:16.648701 24570 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db32a415010a47e995de7ca8d2bbf1ed" candidate_uuid: "2198ba0c75d9450d81673132f3922db8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c3a5167af887498198f016f81be7103f" is_pre_election: true
I20260620 06:09:16.649000 24570 raft_consensus.cc:2454] T db32a415010a47e995de7ca8d2bbf1ed P c3a5167af887498198f016f81be7103f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2198ba0c75d9450d81673132f3922db8 in term 0.
I20260620 06:09:16.649040 24790 leader_election.cc:302] T db32a415010a47e995de7ca8d2bbf1ed P 2198ba0c75d9450d81673132f3922db8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2198ba0c75d9450d81673132f3922db8, 87154d0791e04b8586ac8a57691afadc; no voters:
I20260620 06:09:16.649366 24943 raft_consensus.cc:2781] T db32a415010a47e995de7ca8d2bbf1ed P 2198ba0c75d9450d81673132f3922db8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 06:09:16.649456 24943 raft_consensus.cc:482] T db32a415010a47e995de7ca8d2bbf1ed P 2198ba0c75d9450d81673132f3922db8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 06:09:16.649500 24943 raft_consensus.cc:3037] T db32a415010a47e995de7ca8d2bbf1ed P 2198ba0c75d9450d81673132f3922db8 [term 0 FOLLOWER]: Advancing to term 1
I20260620 06:09:16.650828 24943 raft_consensus.cc:504] T db32a415010a47e995de7ca8d2bbf1ed P 2198ba0c75d9450d81673132f3922db8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87154d0791e04b8586ac8a57691afadc" member_type: VOTER last_known_addr { host: "127.11.102.2" port: 37915 } } peers { permanent_uuid: "c3a5167af887498198f016f81be7103f" member_type: VOTER last_known_addr { host: "127.11.102.1" port: 33233 } } peers { permanent_uuid: "2198ba0c75d9450d81673132f3922db8" member_type: VOTER last_known_addr { host: "127.11.102.3" port: 44981 } }
I20260620 06:09:16.651209 24943 leader_election.cc:288] T db32a415010a47e995de7ca8d2bbf1ed P 2198ba0c75d9450d81673132f3922db8 [CANDIDATE]: Term 1 election: Requested vote from peers 87154d0791e04b8586ac8a57691afadc (127.11.102.2:37915), c3a5167af887498198f016f81be7103f (127.11.102.1:33233)
I20260620 06:09:16.651562 24570 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db32a415010a47e995de7ca8d2bbf1ed" candidate_uuid: "2198ba0c75d9450d81673132f3922db8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c3a5167af887498198f016f81be7103f"
I20260620 06:09:16.651719 24570 raft_consensus.cc:3037] T db32a415010a47e995de7ca8d2bbf1ed P c3a5167af887498198f016f81be7103f [term 0 FOLLOWER]: Advancing to term 1
I20260620 06:09:16.651738 24718 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db32a415010a47e995de7ca8d2bbf1ed" candidate_uuid: "2198ba0c75d9450d81673132f3922db8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87154d0791e04b8586ac8a57691afadc"
I20260620 06:09:16.651870 24718 raft_consensus.cc:3037] T db32a415010a47e995de7ca8d2bbf1ed P 87154d0791e04b8586ac8a57691afadc [term 0 FOLLOWER]: Advancing to term 1
I20260620 06:09:16.652998 24570 raft_consensus.cc:2454] T db32a415010a47e995de7ca8d2bbf1ed P c3a5167af887498198f016f81be7103f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2198ba0c75d9450d81673132f3922db8 in term 1.
I20260620 06:09:16.653057 24718 raft_consensus.cc:2454] T db32a415010a47e995de7ca8d2bbf1ed P 87154d0791e04b8586ac8a57691afadc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2198ba0c75d9450d81673132f3922db8 in term 1.
I20260620 06:09:16.653450 24790 leader_election.cc:302] T db32a415010a47e995de7ca8d2bbf1ed P 2198ba0c75d9450d81673132f3922db8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2198ba0c75d9450d81673132f3922db8, 87154d0791e04b8586ac8a57691afadc; no voters:
I20260620 06:09:16.653662 24943 raft_consensus.cc:2781] T db32a415010a47e995de7ca8d2bbf1ed P 2198ba0c75d9450d81673132f3922db8 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 06:09:16.653889 24943 raft_consensus.cc:686] T db32a415010a47e995de7ca8d2bbf1ed P 2198ba0c75d9450d81673132f3922db8 [term 1 LEADER]: Becoming Leader. State: Replica: 2198ba0c75d9450d81673132f3922db8, State: Running, Role: LEADER
I20260620 06:09:16.654258 24943 consensus_queue.cc:237] T db32a415010a47e995de7ca8d2bbf1ed P 2198ba0c75d9450d81673132f3922db8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87154d0791e04b8586ac8a57691afadc" member_type: VOTER last_known_addr { host: "127.11.102.2" port: 37915 } } peers { permanent_uuid: "c3a5167af887498198f016f81be7103f" member_type: VOTER last_known_addr { host: "127.11.102.1" port: 33233 } } peers { permanent_uuid: "2198ba0c75d9450d81673132f3922db8" member_type: VOTER last_known_addr { host: "127.11.102.3" port: 44981 } }
I20260620 06:09:16.657516 24437 catalog_manager.cc:5495] T db32a415010a47e995de7ca8d2bbf1ed P 2198ba0c75d9450d81673132f3922db8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2198ba0c75d9450d81673132f3922db8 (127.11.102.3). New cstate: current_term: 1 leader_uuid: "2198ba0c75d9450d81673132f3922db8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87154d0791e04b8586ac8a57691afadc" member_type: VOTER last_known_addr { host: "127.11.102.2" port: 37915 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c3a5167af887498198f016f81be7103f" member_type: VOTER last_known_addr { host: "127.11.102.1" port: 33233 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2198ba0c75d9450d81673132f3922db8" member_type: VOTER last_known_addr { host: "127.11.102.3" port: 44981 } health_report { overall_health: HEALTHY } } }
I20260620 06:09:16.671066 24935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:09:16.666145 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43854 (local address 127.11.102.3:44981)
0620 06:09:16.666260 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0620 06:09:16.666266 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 06:09:16.666278 (+ 12us) server_negotiation.cc:407] Connection header received
0620 06:09:16.666354 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:09:16.666356 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:09:16.666407 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:09:16.666525 (+ 118us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 06:09:16.666789 (+ 264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:09:16.667604 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:09:16.668900 (+ 1296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:09:16.669247 (+ 347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:09:16.669344 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:09:16.669772 (+ 428us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 06:09:16.670316 (+ 544us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 06:09:16.670434 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0620 06:09:16.670838 (+ 404us) server_negotiation.cc:299] Negotiation successful
0620 06:09:16.670899 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260620 06:09:16.677991 24718 raft_consensus.cc:1261] T db32a415010a47e995de7ca8d2bbf1ed P 87154d0791e04b8586ac8a57691afadc [term 1 FOLLOWER]: Refusing update from remote peer 2198ba0c75d9450d81673132f3922db8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 06:09:16.677991 24570 raft_consensus.cc:1261] T db32a415010a47e995de7ca8d2bbf1ed P c3a5167af887498198f016f81be7103f [term 1 FOLLOWER]: Refusing update from remote peer 2198ba0c75d9450d81673132f3922db8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 06:09:16.678745 24943 consensus_queue.cc:1035] T db32a415010a47e995de7ca8d2bbf1ed P 2198ba0c75d9450d81673132f3922db8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "87154d0791e04b8586ac8a57691afadc" member_type: VOTER last_known_addr { host: "127.11.102.2" port: 37915 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 06:09:16.679169 24949 consensus_queue.cc:1035] T db32a415010a47e995de7ca8d2bbf1ed P 2198ba0c75d9450d81673132f3922db8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c3a5167af887498198f016f81be7103f" member_type: VOTER last_known_addr { host: "127.11.102.1" port: 33233 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 06:09:16.692533 24952 mvcc.cc:205] Tried to move back new op lower bound from 7298808859345285120 to 7298808859259195392. Current Snapshot: MvccSnapshot[applied={T|T < 7298808859345285120}]
I20260620 06:09:16.696070 24953 mvcc.cc:205] Tried to move back new op lower bound from 7298808859345285120 to 7298808859259195392. Current Snapshot: MvccSnapshot[applied={T|T < 7298808859345285120}]
I20260620 06:09:16.704041 24954 mvcc.cc:205] Tried to move back new op lower bound from 7298808859345285120 to 7298808859259195392. Current Snapshot: MvccSnapshot[applied={T|T < 7298808859345285120}]
I20260620 06:09:16.710361 24436 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55216:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 06:09:16.710568 24436 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55216:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 06:09:16.713025 24436 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f42b7083570d4a95b735450329fb1c2f: Sending DeleteTablet for 3 replicas of tablet db32a415010a47e995de7ca8d2bbf1ed
I20260620 06:09:16.713738 24704 tablet_service.cc:1514] Processing DeleteTablet for tablet db32a415010a47e995de7ca8d2bbf1ed with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 06:09:16 UTC) from {username='kudu', principal='kudu/127.11.102.62@KRBTEST.COM'} at 127.0.0.1:58452
I20260620 06:09:16.713871 24819 tablet_service.cc:1514] Processing DeleteTablet for tablet db32a415010a47e995de7ca8d2bbf1ed with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 06:09:16 UTC) from {username='kudu', principal='kudu/127.11.102.62@KRBTEST.COM'} at 127.0.0.1:43848
I20260620 06:09:16.714397 24962 tablet_replica.cc:330] T db32a415010a47e995de7ca8d2bbf1ed P 87154d0791e04b8586ac8a57691afadc: stopping tablet replica
I20260620 06:09:16.714583 24567 tablet_service.cc:1514] Processing DeleteTablet for tablet db32a415010a47e995de7ca8d2bbf1ed with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 06:09:16 UTC) from {username='kudu', principal='kudu/127.11.102.62@KRBTEST.COM'} at 127.0.0.1:36048
I20260620 06:09:16.714836 24962 raft_consensus.cc:2229] T db32a415010a47e995de7ca8d2bbf1ed P 87154d0791e04b8586ac8a57691afadc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 06:09:16.715185 24962 raft_consensus.cc:2258] T db32a415010a47e995de7ca8d2bbf1ed P 87154d0791e04b8586ac8a57691afadc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 06:09:16.715793 11672 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9HMp43/build/debug/bin/kudu with pid 24475
I20260620 06:09:16.718125 24962 ts_tablet_manager.cc:1905] T db32a415010a47e995de7ca8d2bbf1ed P 87154d0791e04b8586ac8a57691afadc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 06:09:16.721334 24962 ts_tablet_manager.cc:1918] T db32a415010a47e995de7ca8d2bbf1ed P 87154d0791e04b8586ac8a57691afadc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 06:09:16.721498 24962 log.cc:1196] T db32a415010a47e995de7ca8d2bbf1ed P 87154d0791e04b8586ac8a57691afadc: Deleting WAL directory at /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935603120653-11672-0/minicluster-data/ts-1/wal/wals/db32a415010a47e995de7ca8d2bbf1ed
I20260620 06:09:16.721913 24962 ts_tablet_manager.cc:1939] T db32a415010a47e995de7ca8d2bbf1ed P 87154d0791e04b8586ac8a57691afadc: Deleting consensus metadata
I20260620 06:09:16.723240 24421 catalog_manager.cc:4841] TS 87154d0791e04b8586ac8a57691afadc (127.11.102.2:37915): tablet db32a415010a47e995de7ca8d2bbf1ed (table test-table [id=c5cbb8ee9d6c4d46ae8a4234ba353425]) successfully deleted
I20260620 06:09:16.731894 24963 tablet_replica.cc:330] T db32a415010a47e995de7ca8d2bbf1ed P 2198ba0c75d9450d81673132f3922db8: stopping tablet replica
W20260620 06:09:16.732398 24422 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.11.102.1:33233 (error 108)
I20260620 06:09:16.732926 24963 raft_consensus.cc:2229] T db32a415010a47e995de7ca8d2bbf1ed P 2198ba0c75d9450d81673132f3922db8 [term 1 LEADER]: Raft consensus shutting down.
I20260620 06:09:16.733865 24931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:09:16.733548 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.102.1:33233 (local address 127.0.0.1:36064)
0620 06:09:16.733720 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0620 06:09:16.733781 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.102.1:33233: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97}
W20260620 06:09:16.734026 24422 catalog_manager.cc:4584] TS c3a5167af887498198f016f81be7103f (127.11.102.1:33233): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet db32a415010a47e995de7ca8d2bbf1ed: Network error: Client connection negotiation failed: client connection to 127.11.102.1:33233: connect: Connection refused (error 111)
I20260620 06:09:16.734357 11672 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9HMp43/build/debug/bin/kudu with pid 24639
I20260620 06:09:16.735752 24963 raft_consensus.cc:2258] T db32a415010a47e995de7ca8d2bbf1ed P 2198ba0c75d9450d81673132f3922db8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 06:09:16.744099 24963 ts_tablet_manager.cc:1905] T db32a415010a47e995de7ca8d2bbf1ed P 2198ba0c75d9450d81673132f3922db8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 06:09:16.746889 24963 ts_tablet_manager.cc:1918] T db32a415010a47e995de7ca8d2bbf1ed P 2198ba0c75d9450d81673132f3922db8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 06:09:16.746999 24963 log.cc:1196] T db32a415010a47e995de7ca8d2bbf1ed P 2198ba0c75d9450d81673132f3922db8: Deleting WAL directory at /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935603120653-11672-0/minicluster-data/ts-2/wal/wals/db32a415010a47e995de7ca8d2bbf1ed
I20260620 06:09:16.747335 24963 ts_tablet_manager.cc:1939] T db32a415010a47e995de7ca8d2bbf1ed P 2198ba0c75d9450d81673132f3922db8: Deleting consensus metadata
I20260620 06:09:16.748654 24424 catalog_manager.cc:4841] TS 2198ba0c75d9450d81673132f3922db8 (127.11.102.3:44981): tablet db32a415010a47e995de7ca8d2bbf1ed (table test-table [id=c5cbb8ee9d6c4d46ae8a4234ba353425]) successfully deleted
I20260620 06:09:16.749619 11672 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9HMp43/build/debug/bin/kudu with pid 24773
I20260620 06:09:16.756233 11672 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9HMp43/build/debug/bin/kudu with pid 24404
I20260620 06:09:16.762789 11672 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260620 06:09:17.097050 11672 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260620 06:09:17.097184 11672 mini_ranger.cc:66] Stopping Ranger...
I20260620 06:09:17.441865 11672 mini_ranger.cc:68] Stopped Ranger
2026-06-20 06:09:17.442 UTC [23470] LOG: received smart shutdown request
2026-06-20 06:09:17.445 UTC [23470] LOG: background worker "logical replication launcher" (PID 23479) exited with exit code 1
2026-06-20 06:09:17.445 UTC [23474] LOG: shutting down
2026-06-20 06:09:17.585 UTC [23470] LOG: database system is shut down
2026-06-20T06:09:17Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (46620 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781935603120653-11672-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 06:09:17 dist-test-slave-9hm5 krb5kdc[24974](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 06:09:17 dist-test-slave-9hm5 krb5kdc[24974](info): set up 2 sockets
Jun 20 06:09:17 dist-test-slave-9hm5 krb5kdc[24974](info): commencing operation
krb5kdc: starting...
W20260620 06:09:18.743700 11672 mini_kdc.cc:121] Time spent starting KDC: real 1.014s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781935603120653-11672-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781935603120653-11672-0/krb5kdc/test-user.keytab.
Jun 20 06:09:18 dist-test-slave-9hm5 krb5kdc[24974](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935758, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T06:09:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T06:09:18Z Disabled control of system clock
WARNING: no policy specified for kudu/127.11.102.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.102.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.102.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781935603120653-11672-0/krb5kdc/kudu_127.11.102.62.keytab.
Entry for principal kudu/127.11.102.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781935603120653-11672-0/krb5kdc/kudu_127.11.102.62.keytab.
I20260620 06:09:18.866109 11672 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9HMp43/build/debug/bin/kudu
/tmp/dist-test-task9HMp43/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781935603120653-11672-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781935603120653-11672-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781935603120653-11672-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781935603120653-11672-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.102.62:41729
--webserver_interface=127.11.102.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781935603120653-11672-0/krb5kdc/kudu_127.11.102.62.keytab
--principal=kudu/127.11.102.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.102.20:38067
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.102.62:41729,127.11.102.61:34001,127.11.102.60:35001
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781935603120653-11672-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781935603120653-11672-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781935603120653-11672-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 06:09:18.973982 24987 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:09:18.974233 24987 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:09:18.974309 24987 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:09:18.978247 24987 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 06:09:18.978299 24987 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:09:18.978319 24987 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:09:18.978335 24987 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 06:09:18.978350 24987 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 06:09:18.983498 24987 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.102.20:38067
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781935603120653-11672-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781935603120653-11672-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.102.62:41729,127.11.102.61:34001,127.11.102.60:35001
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781935603120653-11672-0/krb5kdc/kudu_127.11.102.62.keytab
--principal=kudu/127.11.102.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.102.62:41729
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781935603120653-11672-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.102.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24987
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781935603120653-11672-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 05:28:48 UTC on ccycloud.quasar-fixhms.root.comops.site
build id 4457020
I20260620 06:09:18.984563 24987 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:09:18.985566 24987 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 06:09:18.993773 24995 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:09:18.993793 24993 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:09:18.993824 24987 server_base.cc:1034] running on GCE node
W20260620 06:09:18.993788 24992 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:09:18.994541 24987 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:09:18.995527 24987 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:09:18.996721 24987 hybrid_clock.cc:648] HybridClock initialized: now 1781935758996706 us; error 33 us; skew 500 ppm
Jun 20 06:09:18 dist-test-slave-9hm5 krb5kdc[24974](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935758, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.102.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 06:09:18.999426 24987 init.cc:377] Logged in from keytab as kudu/127.11.102.62@KRBTEST.COM (short username kudu)
I20260620 06:09:19.000360 24987 webserver.cc:460] Webserver started at http://127.11.102.62:40633/ using document root <none> and password file <none>
I20260620 06:09:19.000905 24987 fs_manager.cc:359] Metadata directory not provided
I20260620 06:09:19.000998 24987 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:09:19.001262 24987 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 06:09:19.003204 24987 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781935603120653-11672-0/minicluster-data/master-0/data/instance:
uuid: "a6625b94c8d1445e8afb60a6c5cfaf22"
format_stamp: "Formatted at 2026-06-20 06:09:18 on dist-test-slave-9hm5"
server_key: "fb82707113eb01a2556ab3dea218891e"
server_key_iv: "489f61a13079c2b9bf06bd80c5288d5e"
server_key_version: "encryptionkey@0"
I20260620 06:09:19.003837 24987 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781935603120653-11672-0/minicluster-data/master-0/wal/instance:
uuid: "a6625b94c8d1445e8afb60a6c5cfaf22"
format_stamp: "Formatted at 2026-06-20 06:09:18 on dist-test-slave-9hm5"
server_key: "fb82707113eb01a2556ab3dea218891e"
server_key_iv: "489f61a13079c2b9bf06bd80c5288d5e"
server_key_version: "encryptionkey@0"
I20260620 06:09:19.007287 24987 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260620 06:09:19.009657 25002 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:09:19.010892 24987 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 06:09:19.011061 24987 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781935603120653-11672-0/minicluster-data/master-0/data,/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781935603120653-11672-0/minicluster-data/master-0/wal
uuid: "a6625b94c8d1445e8afb60a6c5cfaf22"
format_stamp: "Formatted at 2026-06-20 06:09:18 on dist-test-slave-9hm5"
server_key: "fb82707113eb01a2556ab3dea218891e"
server_key_iv: "489f61a13079c2b9bf06bd80c5288d5e"
server_key_version: "encryptionkey@0"
I20260620 06:09:19.011173 24987 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781935603120653-11672-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781935603120653-11672-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781935603120653-11672-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:09:19.022269 24987 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:09:19.024981 24987 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:09:19.025192 24987 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:09:19.032238 24987 rpc_server.cc:307] RPC server started. Bound to: 127.11.102.62:41729
I20260620 06:09:19.032260 25054 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.102.62:41729 every 8 connection(s)
I20260620 06:09:19.033130 24987 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781935603120653-11672-0/minicluster-data/master-0/data/info.pb
I20260620 06:09:19.035830 25055 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 06:09:19.039700 25055 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.11.102.62" port: 41729 } has no permanent_uuid. Determining permanent_uuid...
I20260620 06:09:19.042073 11672 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9HMp43/build/debug/bin/kudu as pid 24987
I20260620 06:09:19.042217 11672 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781935603120653-11672-0/minicluster-data/master-0/wal/instance
I20260620 06:09:19.042572 11672 external_mini_cluster.cc:1428] Setting key d1a85a5b39c12b887f4099f48832a334
Jun 20 06:09:19 dist-test-slave-9hm5 krb5kdc[24974](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935758, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.102.62@KRBTEST.COM for kudu/127.11.102.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.11.102.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.102.61@KRBTEST.COM" created.
I20260620 06:09:19.054025 25057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:09:19.041860 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.102.62:41729 (local address 127.0.0.1:44596)
0620 06:09:19.042434 (+ 574us) negotiation.cc:107] Waiting for socket to connect
0620 06:09:19.042485 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0620 06:09:19.043388 (+ 903us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:09:19.044088 (+ 700us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:09:19.044093 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:09:19.044541 (+ 448us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:09:19.044767 (+ 226us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:09:19.044780 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:09:19.045917 (+ 1137us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:09:19.045920 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:09:19.046628 (+ 708us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:09:19.046637 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:09:19.047266 (+ 629us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:09:19.047270 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:09:19.047407 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:09:19.048130 (+ 723us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:09:19.048145 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:09:19.049523 (+ 1378us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:09:19.051413 (+ 1890us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:09:19.051421 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:09:19.051428 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:09:19.051713 (+ 285us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:09:19.052091 (+ 378us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:09:19.052093 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:09:19.052094 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:09:19.052201 (+ 107us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:09:19.052856 (+ 655us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:09:19.052860 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:09:19.052959 (+ 99us) client_negotiation.cc:769] Sending connection context
0620 06:09:19.053091 (+ 132us) client_negotiation.cc:240] Negotiation successful
0620 06:09:19.053307 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":355,"mutex_wait_us":39,"thread_start_us":149,"threads_started":1}
I20260620 06:09:19.054030 25058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:09:19.042807 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44596 (local address 127.11.102.62:41729)
0620 06:09:19.043119 (+ 312us) server_negotiation.cc:206] Beginning negotiation
0620 06:09:19.043130 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 06:09:19.043173 (+ 43us) server_negotiation.cc:407] Connection header received
0620 06:09:19.043711 (+ 538us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:09:19.043715 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:09:19.043885 (+ 170us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:09:19.044094 (+ 209us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:09:19.044917 (+ 823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:09:19.045773 (+ 856us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:09:19.046749 (+ 976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:09:19.047121 (+ 372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:09:19.047285 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:09:19.049670 (+ 2385us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:09:19.049675 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:09:19.049683 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:09:19.049701 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:09:19.051314 (+ 1613us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:09:19.051834 (+ 520us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:09:19.051839 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:09:19.051840 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:09:19.052005 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:09:19.052285 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:09:19.052287 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:09:19.052288 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:09:19.052682 (+ 394us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:09:19.052823 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0620 06:09:19.053214 (+ 391us) server_negotiation.cc:299] Negotiation successful
0620 06:09:19.053383 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":59,"server-negotiator.queue_time_us":219,"thread_start_us":104,"threads_started":1}
I20260620 06:09:19.056528 25055 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.11.102.61" port: 34001 } has no permanent_uuid. Determining permanent_uuid...
I20260620 06:09:19.057276 25057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:09:19.056974 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.102.61:34001 (local address 127.0.0.1:56642)
0620 06:09:19.057111 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0620 06:09:19.057192 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.102.61:34001: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260620 06:09:19.057507 25005 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.11.102.61:34001: connect: Connection refused (error 111)
I20260620 06:09:19.058368 25057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:09:19.058091 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.102.61:34001 (local address 127.0.0.1:56650)
0620 06:09:19.058225 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0620 06:09:19.058289 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.102.61:34001: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260620 06:09:19.058537 25055 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.11.102.61:34001: Network error: Client connection negotiation failed: client connection to 127.11.102.61:34001: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.102.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781935603120653-11672-0/krb5kdc/kudu_127.11.102.61.keytab.
Entry for principal kudu/127.11.102.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781935603120653-11672-0/krb5kdc/kudu_127.11.102.61.keytab.
I20260620 06:09:19.068481 11672 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9HMp43/build/debug/bin/kudu
/tmp/dist-test-task9HMp43/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781935603120653-11672-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781935603120653-11672-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781935603120653-11672-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781935603120653-11672-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.102.61:34001
--webserver_interface=127.11.102.61
--webserver_port=0
--keytab_file=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781935603120653-11672-0/krb5kdc/kudu_127.11.102.61.keytab
--principal=kudu/127.11.102.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.102.20:38067
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.102.62:41729,127.11.102.61:34001,127.11.102.60:35001
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781935603120653-11672-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781935603120653-11672-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781935603120653-11672-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 06:09:19.107817 25055 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.11.102.61" port: 34001 } attempt: 1
I20260620 06:09:19.108744 25057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:09:19.108434 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.102.61:34001 (local address 127.0.0.1:56664)
0620 06:09:19.108603 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0620 06:09:19.108669 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.102.61:34001: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
I20260620 06:09:19.109534 25057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:09:19.109322 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.102.61:34001 (local address 127.0.0.1:56668)
0620 06:09:19.109429 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0620 06:09:19.109474 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.102.61:34001: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260620 06:09:19.109706 25055 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.11.102.61:34001: Network error: Client connection negotiation failed: client connection to 127.11.102.61:34001: connect: Connection refused (error 111)
W20260620 06:09:19.177047 25062 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:09:19.177289 25062 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:09:19.177328 25062 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260620 06:09:19.177887 25055 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.11.102.61" port: 34001 } attempt: 2
I20260620 06:09:19.178766 25057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:09:19.178465 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.102.61:34001 (local address 127.0.0.1:56676)
0620 06:09:19.178616 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0620 06:09:19.178693 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.102.61:34001: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260620 06:09:19.179493 25057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:09:19.179267 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.102.61:34001 (local address 127.0.0.1:56688)
0620 06:09:19.179391 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0620 06:09:19.179440 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.102.61:34001: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260620 06:09:19.179685 25055 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.11.102.61:34001: Network error: Client connection negotiation failed: client connection to 127.11.102.61:34001: connect: Connection refused (error 111)
W20260620 06:09:19.181918 25062 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 06:09:19.181977 25062 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:09:19.181998 25062 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:09:19.182013 25062 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 06:09:19.182067 25062 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 06:09:19.187330 25062 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.102.20:38067
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781935603120653-11672-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781935603120653-11672-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.11.102.62:41729,127.11.102.61:34001,127.11.102.60:35001
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781935603120653-11672-0/krb5kdc/kudu_127.11.102.61.keytab
--principal=kudu/127.11.102.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.102.61:34001
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781935603120653-11672-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.11.102.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25062
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781935603120653-11672-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 05:28:48 UTC on ccycloud.quasar-fixhms.root.comops.site
build id 4457020
I20260620 06:09:19.188380 25062 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:09:19.189380 25062 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 06:09:19.197638 25070 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:09:19.197690 25068 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:09:19.197636 25067 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:09:19.198057 25062 server_base.cc:1034] running on GCE node
I20260620 06:09:19.198688 25062 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:09:19.199795 25062 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:09:19.200984 25062 hybrid_clock.cc:648] HybridClock initialized: now 1781935759200966 us; error 30 us; skew 500 ppm
Jun 20 06:09:19 dist-test-slave-9hm5 krb5kdc[24974](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935759, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.102.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 06:09:19.203743 25062 init.cc:377] Logged in from keytab as kudu/127.11.102.61@KRBTEST.COM (short username kudu)
I20260620 06:09:19.204656 25062 webserver.cc:460] Webserver started at http://127.11.102.61:44751/ using document root <none> and password file <none>
I20260620 06:09:19.205206 25062 fs_manager.cc:359] Metadata directory not provided
I20260620 06:09:19.205296 25062 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:09:19.205507 25062 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 06:09:19.207433 25062 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781935603120653-11672-0/minicluster-data/master-1/data/instance:
uuid: "dcdaf0ddbf624baabcd4e95370ebb903"
format_stamp: "Formatted at 2026-06-20 06:09:19 on dist-test-slave-9hm5"
server_key: "431819e1d329d9682d2423b18249488e"
server_key_iv: "8c0963a515b1c4abd78106ff88041d61"
server_key_version: "encryptionkey@0"
I20260620 06:09:19.208057 25062 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781935603120653-11672-0/minicluster-data/master-1/wal/instance:
uuid: "dcdaf0ddbf624baabcd4e95370ebb903"
format_stamp: "Formatted at 2026-06-20 06:09:19 on dist-test-slave-9hm5"
server_key: "431819e1d329d9682d2423b18249488e"
server_key_iv: "8c0963a515b1c4abd78106ff88041d61"
server_key_version: "encryptionkey@0"
I20260620 06:09:19.211525 25062 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260620 06:09:19.213886 25077 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:09:19.215152 25062 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 06:09:19.215282 25062 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781935603120653-11672-0/minicluster-data/master-1/data,/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781935603120653-11672-0/minicluster-data/master-1/wal
uuid: "dcdaf0ddbf624baabcd4e95370ebb903"
format_stamp: "Formatted at 2026-06-20 06:09:19 on dist-test-slave-9hm5"
server_key: "431819e1d329d9682d2423b18249488e"
server_key_iv: "8c0963a515b1c4abd78106ff88041d61"
server_key_version: "encryptionkey@0"
I20260620 06:09:19.215389 25062 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781935603120653-11672-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781935603120653-11672-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781935603120653-11672-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:09:19.224749 25062 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:09:19.227500 25062 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:09:19.227720 25062 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:09:19.234974 25062 rpc_server.cc:307] RPC server started. Bound to: 127.11.102.61:34001
I20260620 06:09:19.235000 25129 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.102.61:34001 every 8 connection(s)
I20260620 06:09:19.235957 25062 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781935603120653-11672-0/minicluster-data/master-1/data/info.pb
I20260620 06:09:19.238601 25130 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 06:09:19.242179 25130 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.11.102.62" port: 41729 } has no permanent_uuid. Determining permanent_uuid...
I20260620 06:09:19.244272 11672 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9HMp43/build/debug/bin/kudu as pid 25062
I20260620 06:09:19.244400 11672 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781935603120653-11672-0/minicluster-data/master-1/wal/instance
I20260620 06:09:19.244721 11672 external_mini_cluster.cc:1428] Setting key 693233cbf903f342070e099ba86362a4
Jun 20 06:09:19 dist-test-slave-9hm5 krb5kdc[24974](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935759, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.102.61@KRBTEST.COM for kudu/127.11.102.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.11.102.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.102.60@KRBTEST.COM" created.
I20260620 06:09:19.255702 25058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:09:19.243969 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44602 (local address 127.11.102.62:41729)
0620 06:09:19.244117 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0620 06:09:19.244120 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 06:09:19.245040 (+ 920us) server_negotiation.cc:407] Connection header received
0620 06:09:19.245955 (+ 915us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:09:19.245959 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:09:19.246022 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:09:19.246104 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:09:19.247232 (+ 1128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:09:19.247851 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:09:19.248992 (+ 1141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:09:19.249253 (+ 261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:09:19.249370 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:09:19.252082 (+ 2712us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:09:19.252085 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:09:19.252087 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:09:19.252112 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:09:19.253583 (+ 1471us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:09:19.254096 (+ 513us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:09:19.254104 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:09:19.254106 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:09:19.254180 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:09:19.254522 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:09:19.254527 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:09:19.254528 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:09:19.254740 (+ 212us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:09:19.254843 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0620 06:09:19.255377 (+ 534us) server_negotiation.cc:299] Negotiation successful
0620 06:09:19.255503 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260620 06:09:19.256817 25132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:09:19.244344 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.102.62:41729 (local address 127.0.0.1:44602)
0620 06:09:19.244879 (+ 535us) negotiation.cc:107] Waiting for socket to connect
0620 06:09:19.244922 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0620 06:09:19.245733 (+ 811us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:09:19.246282 (+ 549us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:09:19.246289 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:09:19.246615 (+ 326us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:09:19.247075 (+ 460us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:09:19.247088 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:09:19.247979 (+ 891us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:09:19.247983 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:09:19.248858 (+ 875us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:09:19.248868 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:09:19.249406 (+ 538us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:09:19.249411 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:09:19.249590 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:09:19.250213 (+ 623us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:09:19.250227 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:09:19.251891 (+ 1664us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:09:19.253703 (+ 1812us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:09:19.253709 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:09:19.253725 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:09:19.253983 (+ 258us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:09:19.254273 (+ 290us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:09:19.254277 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:09:19.254279 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:09:19.254418 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:09:19.254855 (+ 437us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:09:19.254863 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:09:19.255123 (+ 260us) client_negotiation.cc:769] Sending connection context
0620 06:09:19.255318 (+ 195us) client_negotiation.cc:240] Negotiation successful
0620 06:09:19.255682 (+ 364us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":300,"thread_start_us":146,"threads_started":1}
I20260620 06:09:19.258302 25130 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.11.102.61" port: 34001 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 06:09:19 dist-test-slave-9hm5 krb5kdc[24974](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935759, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.102.61@KRBTEST.COM for kudu/127.11.102.61@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.102.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781935603120653-11672-0/krb5kdc/kudu_127.11.102.60.keytab.
Entry for principal kudu/127.11.102.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781935603120653-11672-0/krb5kdc/kudu_127.11.102.60.keytab.
I20260620 06:09:19.268133 25132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:09:19.258998 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.102.61:34001 (local address 127.0.0.1:56702)
0620 06:09:19.259156 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0620 06:09:19.259167 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 06:09:19.259230 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:09:19.259809 (+ 579us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:09:19.259812 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:09:19.259989 (+ 177us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:09:19.260059 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:09:19.260064 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:09:19.261013 (+ 949us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:09:19.261019 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:09:19.261552 (+ 533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:09:19.261559 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:09:19.262139 (+ 580us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:09:19.262142 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:09:19.262228 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:09:19.262655 (+ 427us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:09:19.262666 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:09:19.264657 (+ 1991us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:09:19.266524 (+ 1867us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:09:19.266528 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:09:19.266530 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:09:19.266733 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:09:19.267076 (+ 343us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:09:19.267079 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:09:19.267081 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:09:19.267164 (+ 83us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:09:19.267639 (+ 475us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:09:19.267642 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:09:19.267724 (+ 82us) client_negotiation.cc:769] Sending connection context
0620 06:09:19.267841 (+ 117us) client_negotiation.cc:240] Negotiation successful
0620 06:09:19.267970 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":101}
I20260620 06:09:19.268277 25135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:09:19.259123 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56702 (local address 127.11.102.61:34001)
0620 06:09:19.259437 (+ 314us) server_negotiation.cc:206] Beginning negotiation
0620 06:09:19.259446 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 06:09:19.259463 (+ 17us) server_negotiation.cc:407] Connection header received
0620 06:09:19.259493 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:09:19.259497 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:09:19.259699 (+ 202us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:09:19.259833 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:09:19.260165 (+ 332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:09:19.260894 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:09:19.261671 (+ 777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:09:19.262030 (+ 359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:09:19.262142 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:09:19.264799 (+ 2657us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:09:19.264807 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:09:19.264817 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:09:19.264835 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:09:19.266404 (+ 1569us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:09:19.266841 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:09:19.266853 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:09:19.266855 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:09:19.266923 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:09:19.267259 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:09:19.267262 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:09:19.267264 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:09:19.267508 (+ 244us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:09:19.267666 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0620 06:09:19.267954 (+ 288us) server_negotiation.cc:299] Negotiation successful
0620 06:09:19.268083 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":8,"server-negotiator.queue_time_us":225,"thread_start_us":108,"threads_started":1}
I20260620 06:09:19.269104 11672 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9HMp43/build/debug/bin/kudu
/tmp/dist-test-task9HMp43/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781935603120653-11672-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781935603120653-11672-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781935603120653-11672-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781935603120653-11672-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.102.60:35001
--webserver_interface=127.11.102.60
--webserver_port=0
--keytab_file=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781935603120653-11672-0/krb5kdc/kudu_127.11.102.60.keytab
--principal=kudu/127.11.102.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.102.20:38067
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.102.62:41729,127.11.102.61:34001,127.11.102.60:35001
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781935603120653-11672-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781935603120653-11672-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781935603120653-11672-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 06:09:19.270484 25130 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.11.102.60" port: 35001 } has no permanent_uuid. Determining permanent_uuid...
I20260620 06:09:19.270841 25055 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.11.102.61" port: 34001 } attempt: 3
I20260620 06:09:19.271260 25132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:09:19.270995 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.102.60:35001 (local address 127.0.0.1:39582)
0620 06:09:19.271104 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0620 06:09:19.271179 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.102.60:35001: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260620 06:09:19.271551 25079 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.11.102.60:35001: connect: Connection refused (error 111)
I20260620 06:09:19.272898 25132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:09:19.272680 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.102.60:35001 (local address 127.0.0.1:39592)
0620 06:09:19.272792 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0620 06:09:19.272841 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.102.60:35001: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260620 06:09:19.273114 25130 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.11.102.60:35001: Network error: Client connection negotiation failed: client connection to 127.11.102.60:35001: connect: Connection refused (error 111)
Jun 20 06:09:19 dist-test-slave-9hm5 krb5kdc[24974](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935758, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.102.62@KRBTEST.COM for kudu/127.11.102.61@KRBTEST.COM
I20260620 06:09:19.279826 25135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:09:19.271307 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56706 (local address 127.11.102.61:34001)
0620 06:09:19.271404 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0620 06:09:19.271410 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 06:09:19.271521 (+ 111us) server_negotiation.cc:407] Connection header received
0620 06:09:19.271725 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:09:19.271728 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:09:19.271772 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:09:19.271861 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:09:19.272263 (+ 402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:09:19.272945 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:09:19.274203 (+ 1258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:09:19.274421 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:09:19.274528 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:09:19.276838 (+ 2310us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:09:19.276842 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:09:19.276845 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:09:19.276873 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:09:19.278252 (+ 1379us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:09:19.278686 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:09:19.278691 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:09:19.278693 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:09:19.278767 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:09:19.279034 (+ 267us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:09:19.279039 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:09:19.279041 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:09:19.279202 (+ 161us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:09:19.279304 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0620 06:09:19.279570 (+ 266us) server_negotiation.cc:299] Negotiation successful
0620 06:09:19.279705 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260620 06:09:19.279829 25057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:09:19.271271 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.102.61:34001 (local address 127.0.0.1:56706)
0620 06:09:19.271462 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0620 06:09:19.271477 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 06:09:19.271569 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:09:19.271883 (+ 314us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:09:19.271886 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:09:19.272058 (+ 172us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:09:19.272153 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:09:19.272159 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:09:19.273085 (+ 926us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:09:19.273088 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:09:19.274080 (+ 992us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:09:19.274091 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:09:19.274533 (+ 442us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:09:19.274536 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:09:19.274642 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:09:19.275197 (+ 555us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:09:19.275209 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:09:19.276710 (+ 1501us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:09:19.278363 (+ 1653us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:09:19.278367 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:09:19.278369 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:09:19.278573 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:09:19.278857 (+ 284us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:09:19.278861 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:09:19.278863 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:09:19.278946 (+ 83us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:09:19.279306 (+ 360us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:09:19.279311 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:09:19.279406 (+ 95us) client_negotiation.cc:769] Sending connection context
0620 06:09:19.279512 (+ 106us) client_negotiation.cc:240] Negotiation successful
0620 06:09:19.279672 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":106}
I20260620 06:09:19.280849 25055 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.11.102.60" port: 35001 } has no permanent_uuid. Determining permanent_uuid...
I20260620 06:09:19.281486 25057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:09:19.281277 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.102.60:35001 (local address 127.0.0.1:39596)
0620 06:09:19.281385 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0620 06:09:19.281437 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.102.60:35001: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
I20260620 06:09:19.282117 25057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:09:19.281876 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.102.60:35001 (local address 127.0.0.1:39608)
0620 06:09:19.282007 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0620 06:09:19.282055 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.102.60:35001: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
W20260620 06:09:19.282239 25055 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.11.102.60:35001: Network error: Client connection negotiation failed: client connection to 127.11.102.60:35001: connect: Connection refused (error 111)
I20260620 06:09:19.313436 25055 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.11.102.60" port: 35001 } attempt: 1
I20260620 06:09:19.314298 25057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:09:19.314021 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.102.60:35001 (local address 127.0.0.1:39618)
0620 06:09:19.314149 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0620 06:09:19.314223 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.102.60:35001: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260620 06:09:19.315117 25057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:09:19.314833 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.102.60:35001 (local address 127.0.0.1:39634)
0620 06:09:19.314938 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0620 06:09:19.315045 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.102.60:35001: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260620 06:09:19.315356 25055 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.11.102.60:35001: Network error: Client connection negotiation failed: client connection to 127.11.102.60:35001: connect: Connection refused (error 111)
I20260620 06:09:19.322386 25130 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.11.102.60" port: 35001 } attempt: 1
I20260620 06:09:19.323046 25132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:09:19.322795 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.102.60:35001 (local address 127.0.0.1:39640)
0620 06:09:19.322914 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0620 06:09:19.322969 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.102.60:35001: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260620 06:09:19.323767 25132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:09:19.323542 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.102.60:35001 (local address 127.0.0.1:39652)
0620 06:09:19.323661 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0620 06:09:19.323706 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.102.60:35001: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260620 06:09:19.323913 25130 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.11.102.60:35001: Network error: Client connection negotiation failed: client connection to 127.11.102.60:35001: connect: Connection refused (error 111)
W20260620 06:09:19.379518 25136 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:09:19.379875 25136 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:09:19.379927 25136 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:09:19.383913 25136 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 06:09:19.383973 25136 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:09:19.383993 25136 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:09:19.384011 25136 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 06:09:19.384025 25136 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 06:09:19.389310 25136 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.102.20:38067
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781935603120653-11672-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781935603120653-11672-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.11.102.62:41729,127.11.102.61:34001,127.11.102.60:35001
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781935603120653-11672-0/krb5kdc/kudu_127.11.102.60.keytab
--principal=kudu/127.11.102.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.102.60:35001
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781935603120653-11672-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.11.102.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25136
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781935603120653-11672-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 05:28:48 UTC on ccycloud.quasar-fixhms.root.comops.site
build id 4457020
I20260620 06:09:19.390398 25136 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:09:19.390547 25055 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.11.102.60" port: 35001 } attempt: 2
I20260620 06:09:19.391351 25057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:09:19.391022 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.102.60:35001 (local address 127.0.0.1:39668)
0620 06:09:19.391192 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0620 06:09:19.391262 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.102.60:35001: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
I20260620 06:09:19.391688 25136 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 06:09:19.392066 25130 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.11.102.60" port: 35001 } attempt: 2
I20260620 06:09:19.392129 25057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:09:19.391909 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.102.60:35001 (local address 127.0.0.1:39676)
0620 06:09:19.392021 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0620 06:09:19.392073 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.102.60:35001: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260620 06:09:19.392300 25055 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.11.102.60:35001: Network error: Client connection negotiation failed: client connection to 127.11.102.60:35001: connect: Connection refused (error 111)
I20260620 06:09:19.392709 25132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:09:19.392475 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.102.60:35001 (local address 127.0.0.1:39690)
0620 06:09:19.392587 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0620 06:09:19.392640 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.102.60:35001: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260620 06:09:19.393292 25132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:09:19.393091 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.102.60:35001 (local address 127.0.0.1:39692)
0620 06:09:19.393190 (+ 99us) negotiation.cc:107] Waiting for socket to connect
0620 06:09:19.393234 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.102.60:35001: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260620 06:09:19.393433 25130 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.11.102.60:35001: Network error: Client connection negotiation failed: client connection to 127.11.102.60:35001: connect: Connection refused (error 111)
W20260620 06:09:19.400247 25142 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:09:19.400238 25145 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:09:19.400234 25143 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:09:19.400271 25136 server_base.cc:1034] running on GCE node
I20260620 06:09:19.400987 25136 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:09:19.402041 25136 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:09:19.403249 25136 hybrid_clock.cc:648] HybridClock initialized: now 1781935759403221 us; error 46 us; skew 500 ppm
Jun 20 06:09:19 dist-test-slave-9hm5 krb5kdc[24974](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935759, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.102.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 06:09:19.406168 25136 init.cc:377] Logged in from keytab as kudu/127.11.102.60@KRBTEST.COM (short username kudu)
I20260620 06:09:19.407001 25136 webserver.cc:460] Webserver started at http://127.11.102.60:32773/ using document root <none> and password file <none>
I20260620 06:09:19.407493 25136 fs_manager.cc:359] Metadata directory not provided
I20260620 06:09:19.407552 25136 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:09:19.407773 25136 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 06:09:19.409689 25136 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781935603120653-11672-0/minicluster-data/master-2/data/instance:
uuid: "3cb87a0908e94d94aba2daae2c347e49"
format_stamp: "Formatted at 2026-06-20 06:09:19 on dist-test-slave-9hm5"
server_key: "1e9a1c42ae601b53275247fca954a304"
server_key_iv: "53eac819a02e8625a0ad61b9263dde21"
server_key_version: "encryptionkey@0"
I20260620 06:09:19.410193 25136 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781935603120653-11672-0/minicluster-data/master-2/wal/instance:
uuid: "3cb87a0908e94d94aba2daae2c347e49"
format_stamp: "Formatted at 2026-06-20 06:09:19 on dist-test-slave-9hm5"
server_key: "1e9a1c42ae601b53275247fca954a304"
server_key_iv: "53eac819a02e8625a0ad61b9263dde21"
server_key_version: "encryptionkey@0"
I20260620 06:09:19.414626 25136 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260620 06:09:19.417093 25152 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:09:19.418282 25136 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260620 06:09:19.418424 25136 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781935603120653-11672-0/minicluster-data/master-2/data,/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781935603120653-11672-0/minicluster-data/master-2/wal
uuid: "3cb87a0908e94d94aba2daae2c347e49"
format_stamp: "Formatted at 2026-06-20 06:09:19 on dist-test-slave-9hm5"
server_key: "1e9a1c42ae601b53275247fca954a304"
server_key_iv: "53eac819a02e8625a0ad61b9263dde21"
server_key_version: "encryptionkey@0"
I20260620 06:09:19.418548 25136 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781935603120653-11672-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781935603120653-11672-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781935603120653-11672-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:09:19.443889 25136 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:09:19.447098 25136 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:09:19.447259 25136 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:09:19.454281 25136 rpc_server.cc:307] RPC server started. Bound to: 127.11.102.60:35001
I20260620 06:09:19.454298 25204 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.102.60:35001 every 8 connection(s)
I20260620 06:09:19.455235 25136 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781935603120653-11672-0/minicluster-data/master-2/data/info.pb
I20260620 06:09:19.455883 11672 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9HMp43/build/debug/bin/kudu as pid 25136
I20260620 06:09:19.456007 11672 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781935603120653-11672-0/minicluster-data/master-2/wal/instance
I20260620 06:09:19.456326 11672 external_mini_cluster.cc:1428] Setting key 34b03668844a31790d786dd6837e892e
I20260620 06:09:19.458467 25205 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 06:09:19.462662 25205 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.11.102.62" port: 41729 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 06:09:19 dist-test-slave-9hm5 krb5kdc[24974](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935758, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.102.60@KRBTEST.COM
Jun 20 06:09:19 dist-test-slave-9hm5 krb5kdc[24974](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935759, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.102.60@KRBTEST.COM for kudu/127.11.102.62@KRBTEST.COM
I20260620 06:09:19.471068 25208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:09:19.457648 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39696 (local address 127.11.102.60:35001)
0620 06:09:19.458025 (+ 377us) server_negotiation.cc:206] Beginning negotiation
0620 06:09:19.458044 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0620 06:09:19.458089 (+ 45us) server_negotiation.cc:407] Connection header received
0620 06:09:19.458796 (+ 707us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:09:19.458814 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:09:19.459129 (+ 315us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:09:19.459399 (+ 270us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:09:19.460322 (+ 923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:09:19.461328 (+ 1006us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:09:19.462457 (+ 1129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:09:19.462906 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:09:19.463099 (+ 193us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:09:19.465818 (+ 2719us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:09:19.465826 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:09:19.465836 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:09:19.465866 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:09:19.467871 (+ 2005us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:09:19.468355 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:09:19.468359 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:09:19.468363 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:09:19.468469 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:09:19.468788 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:09:19.468794 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:09:19.468798 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:09:19.469404 (+ 606us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:09:19.469610 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0620 06:09:19.469924 (+ 314us) server_negotiation.cc:299] Negotiation successful
0620 06:09:19.470247 (+ 323us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":248,"thread_start_us":135,"threads_started":1}
I20260620 06:09:19.474344 25209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:09:19.464585 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.102.62:41729 (local address 127.0.0.1:44608)
0620 06:09:19.465002 (+ 417us) negotiation.cc:107] Waiting for socket to connect
0620 06:09:19.465051 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0620 06:09:19.465168 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:09:19.465514 (+ 346us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:09:19.465519 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:09:19.465965 (+ 446us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:09:19.466143 (+ 178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:09:19.466155 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:09:19.467076 (+ 921us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:09:19.467080 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:09:19.468062 (+ 982us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:09:19.468069 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:09:19.468571 (+ 502us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:09:19.468577 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:09:19.468680 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:09:19.469180 (+ 500us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:09:19.469194 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:09:19.470729 (+ 1535us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:09:19.472464 (+ 1735us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:09:19.472473 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:09:19.472478 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:09:19.472696 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:09:19.472994 (+ 298us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:09:19.472996 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:09:19.472997 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:09:19.473076 (+ 79us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:09:19.473647 (+ 571us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:09:19.473657 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:09:19.473808 (+ 151us) client_negotiation.cc:769] Sending connection context
0620 06:09:19.473969 (+ 161us) client_negotiation.cc:240] Negotiation successful
0620 06:09:19.474164 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":263,"thread_start_us":181,"threads_started":1}
I20260620 06:09:19.474404 25058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:09:19.464646 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44608 (local address 127.11.102.62:41729)
0620 06:09:19.464796 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0620 06:09:19.464801 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 06:09:19.465137 (+ 336us) server_negotiation.cc:407] Connection header received
0620 06:09:19.465312 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:09:19.465315 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:09:19.465368 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:09:19.465461 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:09:19.466281 (+ 820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:09:19.466959 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:09:19.468164 (+ 1205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:09:19.468461 (+ 297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:09:19.468564 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:09:19.470881 (+ 2317us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:09:19.470884 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:09:19.470886 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:09:19.470910 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:09:19.472337 (+ 1427us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:09:19.472798 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:09:19.472803 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:09:19.472805 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:09:19.472896 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:09:19.473270 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:09:19.473274 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:09:19.473276 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:09:19.473516 (+ 240us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:09:19.473638 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0620 06:09:19.474078 (+ 440us) server_negotiation.cc:299] Negotiation successful
0620 06:09:19.474200 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260620 06:09:19.475473 25205 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.11.102.61" port: 34001 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 06:09:19 dist-test-slave-9hm5 krb5kdc[24974](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935758, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.102.62@KRBTEST.COM
Jun 20 06:09:19 dist-test-slave-9hm5 krb5kdc[24974](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935759, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.102.60@KRBTEST.COM for kudu/127.11.102.61@KRBTEST.COM
I20260620 06:09:19.483019 25210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:09:19.473450 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44614 (local address 127.11.102.62:41729)
0620 06:09:19.473863 (+ 413us) server_negotiation.cc:206] Beginning negotiation
0620 06:09:19.473872 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 06:09:19.473885 (+ 13us) server_negotiation.cc:407] Connection header received
0620 06:09:19.473920 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:09:19.473926 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:09:19.473978 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:09:19.474082 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:09:19.474874 (+ 792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:09:19.475595 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:09:19.476560 (+ 965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:09:19.476823 (+ 263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:09:19.476934 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:09:19.479513 (+ 2579us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:09:19.479517 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:09:19.479519 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:09:19.479550 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:09:19.481287 (+ 1737us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:09:19.481754 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:09:19.481758 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:09:19.481759 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:09:19.481853 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:09:19.482161 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:09:19.482167 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:09:19.482172 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:09:19.482399 (+ 227us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:09:19.482612 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0620 06:09:19.482693 (+ 81us) server_negotiation.cc:299] Negotiation successful
0620 06:09:19.482810 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":313,"thread_start_us":130,"threads_started":1}
I20260620 06:09:19.484575 25130 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.11.102.60" port: 35001 } attempt: 3
I20260620 06:09:19.484619 25135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:09:19.476158 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56712 (local address 127.11.102.61:34001)
0620 06:09:19.476301 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0620 06:09:19.476306 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 06:09:19.476321 (+ 15us) server_negotiation.cc:407] Connection header received
0620 06:09:19.476389 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:09:19.476392 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:09:19.476435 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:09:19.476525 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:09:19.476907 (+ 382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:09:19.477562 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:09:19.478309 (+ 747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:09:19.478642 (+ 333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:09:19.478757 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:09:19.481058 (+ 2301us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:09:19.481064 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:09:19.481068 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:09:19.481099 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:09:19.482763 (+ 1664us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:09:19.483238 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:09:19.483242 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:09:19.483244 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:09:19.483327 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:09:19.483643 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:09:19.483647 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:09:19.483648 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:09:19.483868 (+ 220us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:09:19.483973 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0620 06:09:19.484342 (+ 369us) server_negotiation.cc:299] Negotiation successful
0620 06:09:19.484472 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260620 06:09:19.484975 25209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:09:19.476049 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.102.61:34001 (local address 127.0.0.1:56712)
0620 06:09:19.476221 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0620 06:09:19.476235 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 06:09:19.476304 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:09:19.476538 (+ 234us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:09:19.476541 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:09:19.476705 (+ 164us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:09:19.476793 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:09:19.476798 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:09:19.477682 (+ 884us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:09:19.477685 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:09:19.478192 (+ 507us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:09:19.478199 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:09:19.478772 (+ 573us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:09:19.478775 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:09:19.478858 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:09:19.479444 (+ 586us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:09:19.479456 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:09:19.480915 (+ 1459us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:09:19.482885 (+ 1970us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:09:19.482889 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:09:19.482891 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:09:19.483118 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:09:19.483428 (+ 310us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:09:19.483431 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:09:19.483433 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:09:19.483515 (+ 82us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:09:19.484002 (+ 487us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:09:19.484005 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:09:19.484116 (+ 111us) client_negotiation.cc:769] Sending connection context
0620 06:09:19.484669 (+ 553us) client_negotiation.cc:240] Negotiation successful
0620 06:09:19.484814 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":97}
I20260620 06:09:19.485833 25205 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.11.102.60" port: 35001 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 06:09:19 dist-test-slave-9hm5 krb5kdc[24974](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935759, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.102.61@KRBTEST.COM for kudu/127.11.102.60@KRBTEST.COM
Jun 20 06:09:19 dist-test-slave-9hm5 krb5kdc[24974](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935758, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.102.61@KRBTEST.COM
Jun 20 06:09:19 dist-test-slave-9hm5 krb5kdc[24974](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935759, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.102.60@KRBTEST.COM for kudu/127.11.102.60@KRBTEST.COM
I20260620 06:09:19.491479 25055 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.11.102.60" port: 35001 } attempt: 3
I20260620 06:09:19.494422 25208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:09:19.485019 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39710 (local address 127.11.102.60:35001)
0620 06:09:19.485330 (+ 311us) server_negotiation.cc:206] Beginning negotiation
0620 06:09:19.485334 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 06:09:19.485345 (+ 11us) server_negotiation.cc:407] Connection header received
0620 06:09:19.485373 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:09:19.485376 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:09:19.485416 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:09:19.485508 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:09:19.485867 (+ 359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:09:19.486680 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:09:19.487728 (+ 1048us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:09:19.488077 (+ 349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:09:19.488167 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:09:19.490508 (+ 2341us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:09:19.490514 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:09:19.490519 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:09:19.490541 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:09:19.492254 (+ 1713us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:09:19.492803 (+ 549us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:09:19.492807 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:09:19.492809 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:09:19.492887 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:09:19.493191 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:09:19.493196 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:09:19.493201 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:09:19.493420 (+ 219us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:09:19.493523 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0620 06:09:19.494107 (+ 584us) server_negotiation.cc:299] Negotiation successful
0620 06:09:19.494266 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":242}
I20260620 06:09:19.495252 25211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:09:19.484244 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56714 (local address 127.11.102.61:34001)
0620 06:09:19.484599 (+ 355us) server_negotiation.cc:206] Beginning negotiation
0620 06:09:19.484607 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 06:09:19.484620 (+ 13us) server_negotiation.cc:407] Connection header received
0620 06:09:19.484658 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:09:19.484661 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:09:19.484701 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:09:19.484794 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:09:19.486116 (+ 1322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:09:19.486816 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:09:19.487724 (+ 908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:09:19.488071 (+ 347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:09:19.488168 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:09:19.490930 (+ 2762us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:09:19.490955 (+ 25us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:09:19.490957 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:09:19.490982 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:09:19.493116 (+ 2134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:09:19.493610 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:09:19.493613 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:09:19.493615 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:09:19.493697 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:09:19.493992 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:09:19.493995 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:09:19.493997 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:09:19.494220 (+ 223us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:09:19.494400 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0620 06:09:19.494967 (+ 567us) server_negotiation.cc:299] Negotiation successful
0620 06:09:19.495087 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":291,"thread_start_us":113,"threads_started":1}
I20260620 06:09:19.495445 25132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:09:19.484953 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.102.60:35001 (local address 127.0.0.1:39710)
0620 06:09:19.485095 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0620 06:09:19.485112 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 06:09:19.485186 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:09:19.485524 (+ 338us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:09:19.485527 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:09:19.485680 (+ 153us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:09:19.485769 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:09:19.485777 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:09:19.486796 (+ 1019us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:09:19.486799 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:09:19.487566 (+ 767us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:09:19.487574 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:09:19.488189 (+ 615us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:09:19.488193 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:09:19.488279 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:09:19.488771 (+ 492us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:09:19.488787 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:09:19.490050 (+ 1263us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:09:19.492364 (+ 2314us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:09:19.492367 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:09:19.492369 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:09:19.492689 (+ 320us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:09:19.492984 (+ 295us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:09:19.492990 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:09:19.492993 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:09:19.493077 (+ 84us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:09:19.493573 (+ 496us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:09:19.493578 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:09:19.493688 (+ 110us) client_negotiation.cc:769] Sending connection context
0620 06:09:19.493796 (+ 108us) client_negotiation.cc:240] Negotiation successful
0620 06:09:19.493930 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":59}
I20260620 06:09:19.496672 25212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:09:19.486837 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39712 (local address 127.11.102.60:35001)
0620 06:09:19.487182 (+ 345us) server_negotiation.cc:206] Beginning negotiation
0620 06:09:19.487190 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 06:09:19.487202 (+ 12us) server_negotiation.cc:407] Connection header received
0620 06:09:19.487233 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:09:19.487235 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:09:19.487269 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:09:19.487352 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:09:19.487784 (+ 432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:09:19.488415 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:09:19.489305 (+ 890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:09:19.489753 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:09:19.489850 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:09:19.492920 (+ 3070us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:09:19.492925 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:09:19.492940 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:09:19.492962 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:09:19.494698 (+ 1736us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:09:19.495147 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:09:19.495154 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:09:19.495156 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:09:19.495236 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:09:19.495777 (+ 541us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:09:19.495780 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:09:19.495782 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:09:19.495997 (+ 215us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:09:19.496065 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0620 06:09:19.496376 (+ 311us) server_negotiation.cc:299] Negotiation successful
0620 06:09:19.496507 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":29,"server-negotiator.queue_time_us":285,"thread_start_us":114,"threads_started":1}
I20260620 06:09:19.497080 25209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:09:19.486276 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.102.60:35001 (local address 127.0.0.1:39712)
0620 06:09:19.486373 (+ 97us) negotiation.cc:107] Waiting for socket to connect
0620 06:09:19.486385 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 06:09:19.486450 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:09:19.487393 (+ 943us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:09:19.487396 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:09:19.487551 (+ 155us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:09:19.487692 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:09:19.487702 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:09:19.488499 (+ 797us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:09:19.488505 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:09:19.489167 (+ 662us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:09:19.489179 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:09:19.489854 (+ 675us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:09:19.489858 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:09:19.489974 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:09:19.490414 (+ 440us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:09:19.490442 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:09:19.492803 (+ 2361us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:09:19.494810 (+ 2007us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:09:19.494813 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:09:19.494815 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:09:19.495054 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:09:19.495577 (+ 523us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:09:19.495581 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:09:19.495585 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:09:19.495691 (+ 106us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:09:19.496123 (+ 432us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:09:19.496126 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:09:19.496207 (+ 81us) client_negotiation.cc:769] Sending connection context
0620 06:09:19.496861 (+ 654us) client_negotiation.cc:240] Negotiation successful
0620 06:09:19.496986 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32,"mutex_wait_us":2}
Jun 20 06:09:19 dist-test-slave-9hm5 krb5kdc[24974](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935758, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.102.62@KRBTEST.COM for kudu/127.11.102.60@KRBTEST.COM
I20260620 06:09:19.499588 25130 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dcdaf0ddbf624baabcd4e95370ebb903: Bootstrap starting.
I20260620 06:09:19.500082 25205 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3cb87a0908e94d94aba2daae2c347e49: Bootstrap starting.
I20260620 06:09:19.501590 25057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:09:19.491897 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.102.60:35001 (local address 127.0.0.1:39716)
0620 06:09:19.492156 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0620 06:09:19.492171 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 06:09:19.492247 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:09:19.492714 (+ 467us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:09:19.492717 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:09:19.492896 (+ 179us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:09:19.493008 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:09:19.493015 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:09:19.494645 (+ 1630us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:09:19.494649 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:09:19.495408 (+ 759us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:09:19.495419 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:09:19.496210 (+ 791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:09:19.496213 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:09:19.496300 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:09:19.496745 (+ 445us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:09:19.496764 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:09:19.498147 (+ 1383us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:09:19.500081 (+ 1934us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:09:19.500083 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:09:19.500085 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:09:19.500323 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:09:19.500609 (+ 286us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:09:19.500612 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:09:19.500614 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:09:19.500706 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:09:19.501111 (+ 405us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:09:19.501114 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:09:19.501212 (+ 98us) client_negotiation.cc:769] Sending connection context
0620 06:09:19.501318 (+ 106us) client_negotiation.cc:240] Negotiation successful
0620 06:09:19.501443 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":176}
I20260620 06:09:19.501644 25213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:09:19.492159 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39716 (local address 127.11.102.60:35001)
0620 06:09:19.492428 (+ 269us) server_negotiation.cc:206] Beginning negotiation
0620 06:09:19.492438 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 06:09:19.492453 (+ 15us) server_negotiation.cc:407] Connection header received
0620 06:09:19.492492 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:09:19.492498 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:09:19.492545 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:09:19.493125 (+ 580us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:09:19.493535 (+ 410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:09:19.494501 (+ 966us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:09:19.495890 (+ 1389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:09:19.496092 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:09:19.496195 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:09:19.498281 (+ 2086us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:09:19.498285 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:09:19.498287 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:09:19.498309 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:09:19.499963 (+ 1654us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:09:19.500426 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:09:19.500429 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:09:19.500431 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:09:19.500513 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:09:19.500802 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:09:19.500805 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:09:19.500806 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:09:19.501010 (+ 204us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:09:19.501108 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0620 06:09:19.501388 (+ 280us) server_negotiation.cc:299] Negotiation successful
0620 06:09:19.501492 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":198,"thread_start_us":119,"threads_started":1}
I20260620 06:09:19.502449 25130 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P dcdaf0ddbf624baabcd4e95370ebb903: Neither blocks nor log segments found. Creating new log.
I20260620 06:09:19.502661 25205 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3cb87a0908e94d94aba2daae2c347e49: Neither blocks nor log segments found. Creating new log.
I20260620 06:09:19.503239 25130 log.cc:824] T 00000000000000000000000000000000 P dcdaf0ddbf624baabcd4e95370ebb903: Log is configured to *not* fsync() on all Append() calls
I20260620 06:09:19.503470 25205 log.cc:824] T 00000000000000000000000000000000 P 3cb87a0908e94d94aba2daae2c347e49: Log is configured to *not* fsync() on all Append() calls
I20260620 06:09:19.504756 25055 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a6625b94c8d1445e8afb60a6c5cfaf22: Bootstrap starting.
I20260620 06:09:19.505617 25205 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3cb87a0908e94d94aba2daae2c347e49: No bootstrap required, opened a new log
I20260620 06:09:19.505617 25130 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dcdaf0ddbf624baabcd4e95370ebb903: No bootstrap required, opened a new log
I20260620 06:09:19.506698 25055 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a6625b94c8d1445e8afb60a6c5cfaf22: Neither blocks nor log segments found. Creating new log.
I20260620 06:09:19.507287 25055 log.cc:824] T 00000000000000000000000000000000 P a6625b94c8d1445e8afb60a6c5cfaf22: Log is configured to *not* fsync() on all Append() calls
I20260620 06:09:19.509079 11672 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 06:09:19.509699 25055 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a6625b94c8d1445e8afb60a6c5cfaf22: No bootstrap required, opened a new log
I20260620 06:09:19.509729 25130 raft_consensus.cc:348] T 00000000000000000000000000000000 P dcdaf0ddbf624baabcd4e95370ebb903 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6625b94c8d1445e8afb60a6c5cfaf22" member_type: VOTER last_known_addr { host: "127.11.102.62" port: 41729 } } peers { permanent_uuid: "dcdaf0ddbf624baabcd4e95370ebb903" member_type: VOTER last_known_addr { host: "127.11.102.61" port: 34001 } } peers { permanent_uuid: "3cb87a0908e94d94aba2daae2c347e49" member_type: VOTER last_known_addr { host: "127.11.102.60" port: 35001 } }
I20260620 06:09:19.509951 25130 raft_consensus.cc:374] T 00000000000000000000000000000000 P dcdaf0ddbf624baabcd4e95370ebb903 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 06:09:19.509999 25130 raft_consensus.cc:729] T 00000000000000000000000000000000 P dcdaf0ddbf624baabcd4e95370ebb903 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dcdaf0ddbf624baabcd4e95370ebb903, State: Initialized, Role: FOLLOWER
I20260620 06:09:19.510300 25205 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3cb87a0908e94d94aba2daae2c347e49 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6625b94c8d1445e8afb60a6c5cfaf22" member_type: VOTER last_known_addr { host: "127.11.102.62" port: 41729 } } peers { permanent_uuid: "dcdaf0ddbf624baabcd4e95370ebb903" member_type: VOTER last_known_addr { host: "127.11.102.61" port: 34001 } } peers { permanent_uuid: "3cb87a0908e94d94aba2daae2c347e49" member_type: VOTER last_known_addr { host: "127.11.102.60" port: 35001 } }
I20260620 06:09:19.510505 25205 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3cb87a0908e94d94aba2daae2c347e49 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 06:09:19.510550 25205 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3cb87a0908e94d94aba2daae2c347e49 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3cb87a0908e94d94aba2daae2c347e49, State: Initialized, Role: FOLLOWER
I20260620 06:09:19.510480 25130 consensus_queue.cc:260] T 00000000000000000000000000000000 P dcdaf0ddbf624baabcd4e95370ebb903 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6625b94c8d1445e8afb60a6c5cfaf22" member_type: VOTER last_known_addr { host: "127.11.102.62" port: 41729 } } peers { permanent_uuid: "dcdaf0ddbf624baabcd4e95370ebb903" member_type: VOTER last_known_addr { host: "127.11.102.61" port: 34001 } } peers { permanent_uuid: "3cb87a0908e94d94aba2daae2c347e49" member_type: VOTER last_known_addr { host: "127.11.102.60" port: 35001 } }
I20260620 06:09:19.510962 25205 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3cb87a0908e94d94aba2daae2c347e49 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6625b94c8d1445e8afb60a6c5cfaf22" member_type: VOTER last_known_addr { host: "127.11.102.62" port: 41729 } } peers { permanent_uuid: "dcdaf0ddbf624baabcd4e95370ebb903" member_type: VOTER last_known_addr { host: "127.11.102.61" port: 34001 } } peers { permanent_uuid: "3cb87a0908e94d94aba2daae2c347e49" member_type: VOTER last_known_addr { host: "127.11.102.60" port: 35001 } }
I20260620 06:09:19.511830 25130 sys_catalog.cc:564] T 00000000000000000000000000000000 P dcdaf0ddbf624baabcd4e95370ebb903 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 06:09:19.512223 25205 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3cb87a0908e94d94aba2daae2c347e49 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 06:09:19.513345 25221 sys_catalog.cc:455] T 00000000000000000000000000000000 P dcdaf0ddbf624baabcd4e95370ebb903 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6625b94c8d1445e8afb60a6c5cfaf22" member_type: VOTER last_known_addr { host: "127.11.102.62" port: 41729 } } peers { permanent_uuid: "dcdaf0ddbf624baabcd4e95370ebb903" member_type: VOTER last_known_addr { host: "127.11.102.61" port: 34001 } } peers { permanent_uuid: "3cb87a0908e94d94aba2daae2c347e49" member_type: VOTER last_known_addr { host: "127.11.102.60" port: 35001 } } }
I20260620 06:09:19.513527 25221 sys_catalog.cc:458] T 00000000000000000000000000000000 P dcdaf0ddbf624baabcd4e95370ebb903 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 06:09:19.513814 25222 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3cb87a0908e94d94aba2daae2c347e49 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6625b94c8d1445e8afb60a6c5cfaf22" member_type: VOTER last_known_addr { host: "127.11.102.62" port: 41729 } } peers { permanent_uuid: "dcdaf0ddbf624baabcd4e95370ebb903" member_type: VOTER last_known_addr { host: "127.11.102.61" port: 34001 } } peers { permanent_uuid: "3cb87a0908e94d94aba2daae2c347e49" member_type: VOTER last_known_addr { host: "127.11.102.60" port: 35001 } } }
I20260620 06:09:19.513973 25222 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3cb87a0908e94d94aba2daae2c347e49 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 06:09:19.513898 25055 raft_consensus.cc:348] T 00000000000000000000000000000000 P a6625b94c8d1445e8afb60a6c5cfaf22 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6625b94c8d1445e8afb60a6c5cfaf22" member_type: VOTER last_known_addr { host: "127.11.102.62" port: 41729 } } peers { permanent_uuid: "dcdaf0ddbf624baabcd4e95370ebb903" member_type: VOTER last_known_addr { host: "127.11.102.61" port: 34001 } } peers { permanent_uuid: "3cb87a0908e94d94aba2daae2c347e49" member_type: VOTER last_known_addr { host: "127.11.102.60" port: 35001 } }
I20260620 06:09:19.514259 25055 raft_consensus.cc:374] T 00000000000000000000000000000000 P a6625b94c8d1445e8afb60a6c5cfaf22 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 06:09:19.514369 25055 raft_consensus.cc:729] T 00000000000000000000000000000000 P a6625b94c8d1445e8afb60a6c5cfaf22 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a6625b94c8d1445e8afb60a6c5cfaf22, State: Initialized, Role: FOLLOWER
W20260620 06:09:19.518215 25244 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 3cb87a0908e94d94aba2daae2c347e49: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 06:09:19.518244 25247 catalog_manager.cc:1558] T 00000000000000000000000000000000 P dcdaf0ddbf624baabcd4e95370ebb903: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 06:09:19.518306 25247 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20260620 06:09:19.518383 25244 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 06:09:19.515970 25136 master_runner.cc:186] Error getting master registration for 127.11.102.62:41729: OK, instance_id { permanent_uuid: "a6625b94c8d1445e8afb60a6c5cfaf22" instance_seqno: 1781935759030162 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260620 06:09:19.516193 25062 master_runner.cc:186] Error getting master registration for 127.11.102.62:41729: OK, instance_id { permanent_uuid: "a6625b94c8d1445e8afb60a6c5cfaf22" instance_seqno: 1781935759030162 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260620 06:09:19.519181 25136 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260620 06:09:19.519205 25062 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260620 06:09:19.514864 25055 consensus_queue.cc:260] T 00000000000000000000000000000000 P a6625b94c8d1445e8afb60a6c5cfaf22 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6625b94c8d1445e8afb60a6c5cfaf22" member_type: VOTER last_known_addr { host: "127.11.102.62" port: 41729 } } peers { permanent_uuid: "dcdaf0ddbf624baabcd4e95370ebb903" member_type: VOTER last_known_addr { host: "127.11.102.61" port: 34001 } } peers { permanent_uuid: "3cb87a0908e94d94aba2daae2c347e49" member_type: VOTER last_known_addr { host: "127.11.102.60" port: 35001 } }
I20260620 06:09:19.519435 25222 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3cb87a0908e94d94aba2daae2c347e49 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 06:09:19.519583 25222 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3cb87a0908e94d94aba2daae2c347e49 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6625b94c8d1445e8afb60a6c5cfaf22" member_type: VOTER last_known_addr { host: "127.11.102.62" port: 41729 } } peers { permanent_uuid: "dcdaf0ddbf624baabcd4e95370ebb903" member_type: VOTER last_known_addr { host: "127.11.102.61" port: 34001 } } peers { permanent_uuid: "3cb87a0908e94d94aba2daae2c347e49" member_type: VOTER last_known_addr { host: "127.11.102.60" port: 35001 } }
I20260620 06:09:19.520531 25222 leader_election.cc:288] T 00000000000000000000000000000000 P 3cb87a0908e94d94aba2daae2c347e49 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a6625b94c8d1445e8afb60a6c5cfaf22 (127.11.102.62:41729), dcdaf0ddbf624baabcd4e95370ebb903 (127.11.102.61:34001)
I20260620 06:09:19.521138 25104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "3cb87a0908e94d94aba2daae2c347e49" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dcdaf0ddbf624baabcd4e95370ebb903" is_pre_election: true
I20260620 06:09:19.521420 25104 raft_consensus.cc:2454] T 00000000000000000000000000000000 P dcdaf0ddbf624baabcd4e95370ebb903 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3cb87a0908e94d94aba2daae2c347e49 in term 0.
I20260620 06:09:19.521849 25155 leader_election.cc:302] T 00000000000000000000000000000000 P 3cb87a0908e94d94aba2daae2c347e49 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3cb87a0908e94d94aba2daae2c347e49, dcdaf0ddbf624baabcd4e95370ebb903; no voters:
I20260620 06:09:19.521953 25055 sys_catalog.cc:564] T 00000000000000000000000000000000 P a6625b94c8d1445e8afb60a6c5cfaf22 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 06:09:19.522202 25029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "3cb87a0908e94d94aba2daae2c347e49" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a6625b94c8d1445e8afb60a6c5cfaf22" is_pre_election: true
I20260620 06:09:19.522455 25029 raft_consensus.cc:2454] T 00000000000000000000000000000000 P a6625b94c8d1445e8afb60a6c5cfaf22 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3cb87a0908e94d94aba2daae2c347e49 in term 0.
I20260620 06:09:19.523125 25222 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3cb87a0908e94d94aba2daae2c347e49 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 06:09:19.523234 25222 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3cb87a0908e94d94aba2daae2c347e49 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 06:09:19.523280 25222 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3cb87a0908e94d94aba2daae2c347e49 [term 0 FOLLOWER]: Advancing to term 1
I20260620 06:09:19.525278 25222 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3cb87a0908e94d94aba2daae2c347e49 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6625b94c8d1445e8afb60a6c5cfaf22" member_type: VOTER last_known_addr { host: "127.11.102.62" port: 41729 } } peers { permanent_uuid: "dcdaf0ddbf624baabcd4e95370ebb903" member_type: VOTER last_known_addr { host: "127.11.102.61" port: 34001 } } peers { permanent_uuid: "3cb87a0908e94d94aba2daae2c347e49" member_type: VOTER last_known_addr { host: "127.11.102.60" port: 35001 } }
I20260620 06:09:19.525928 25029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "3cb87a0908e94d94aba2daae2c347e49" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a6625b94c8d1445e8afb60a6c5cfaf22"
I20260620 06:09:19.526041 25029 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a6625b94c8d1445e8afb60a6c5cfaf22 [term 0 FOLLOWER]: Advancing to term 1
W20260620 06:09:19.527133 25008 tablet.cc:2369] T 00000000000000000000000000000000 P a6625b94c8d1445e8afb60a6c5cfaf22: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 06:09:19.527350 25029 raft_consensus.cc:2454] T 00000000000000000000000000000000 P a6625b94c8d1445e8afb60a6c5cfaf22 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3cb87a0908e94d94aba2daae2c347e49 in term 1.
I20260620 06:09:19.527854 25156 leader_election.cc:302] T 00000000000000000000000000000000 P 3cb87a0908e94d94aba2daae2c347e49 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3cb87a0908e94d94aba2daae2c347e49, a6625b94c8d1445e8afb60a6c5cfaf22; no voters:
I20260620 06:09:19.528473 25222 leader_election.cc:288] T 00000000000000000000000000000000 P 3cb87a0908e94d94aba2daae2c347e49 [CANDIDATE]: Term 1 election: Requested vote from peers a6625b94c8d1445e8afb60a6c5cfaf22 (127.11.102.62:41729), dcdaf0ddbf624baabcd4e95370ebb903 (127.11.102.61:34001)
I20260620 06:09:19.528620 25222 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3cb87a0908e94d94aba2daae2c347e49 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 06:09:19.528731 25222 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3cb87a0908e94d94aba2daae2c347e49 [term 1 LEADER]: Becoming Leader. State: Replica: 3cb87a0908e94d94aba2daae2c347e49, State: Running, Role: LEADER
I20260620 06:09:19.528903 25104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "3cb87a0908e94d94aba2daae2c347e49" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dcdaf0ddbf624baabcd4e95370ebb903"
I20260620 06:09:19.529026 25104 raft_consensus.cc:3037] T 00000000000000000000000000000000 P dcdaf0ddbf624baabcd4e95370ebb903 [term 0 FOLLOWER]: Advancing to term 1
I20260620 06:09:19.529196 25222 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3cb87a0908e94d94aba2daae2c347e49 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6625b94c8d1445e8afb60a6c5cfaf22" member_type: VOTER last_known_addr { host: "127.11.102.62" port: 41729 } } peers { permanent_uuid: "dcdaf0ddbf624baabcd4e95370ebb903" member_type: VOTER last_known_addr { host: "127.11.102.61" port: 34001 } } peers { permanent_uuid: "3cb87a0908e94d94aba2daae2c347e49" member_type: VOTER last_known_addr { host: "127.11.102.60" port: 35001 } }
I20260620 06:09:19.530264 25104 raft_consensus.cc:2454] T 00000000000000000000000000000000 P dcdaf0ddbf624baabcd4e95370ebb903 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3cb87a0908e94d94aba2daae2c347e49 in term 1.
I20260620 06:09:19.530370 25210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:09:19.512044 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44630 (local address 127.11.102.62:41729)
0620 06:09:19.512159 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0620 06:09:19.512164 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 06:09:19.519746 (+ 7582us) server_negotiation.cc:407] Connection header received
0620 06:09:19.519882 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:09:19.519885 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:09:19.519934 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:09:19.520014 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:09:19.521028 (+ 1014us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:09:19.521857 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:09:19.523048 (+ 1191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:09:19.523418 (+ 370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:09:19.523499 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:09:19.525424 (+ 1925us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:09:19.525428 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:09:19.525431 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:09:19.525456 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:09:19.528433 (+ 2977us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:09:19.528951 (+ 518us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:09:19.528955 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:09:19.528957 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:09:19.529046 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:09:19.529382 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:09:19.529386 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:09:19.529388 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:09:19.529600 (+ 212us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:09:19.529702 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0620 06:09:19.530088 (+ 386us) server_negotiation.cc:299] Negotiation successful
0620 06:09:19.530215 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260620 06:09:19.534309 24987 master_runner.cc:232] Existing masters have differing terms: 0 vs 1
I20260620 06:09:19.534443 24987 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260620 06:09:19.534605 25222 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3cb87a0908e94d94aba2daae2c347e49 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3cb87a0908e94d94aba2daae2c347e49. Latest consensus state: current_term: 1 leader_uuid: "3cb87a0908e94d94aba2daae2c347e49" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6625b94c8d1445e8afb60a6c5cfaf22" member_type: VOTER last_known_addr { host: "127.11.102.62" port: 41729 } } peers { permanent_uuid: "dcdaf0ddbf624baabcd4e95370ebb903" member_type: VOTER last_known_addr { host: "127.11.102.61" port: 34001 } } peers { permanent_uuid: "3cb87a0908e94d94aba2daae2c347e49" member_type: VOTER last_known_addr { host: "127.11.102.60" port: 35001 } } }
I20260620 06:09:19.534773 25222 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3cb87a0908e94d94aba2daae2c347e49 [sys.catalog]: This master's current role is: LEADER
I20260620 06:09:19.535225 25264 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 06:09:19.535383 25250 sys_catalog.cc:455] T 00000000000000000000000000000000 P a6625b94c8d1445e8afb60a6c5cfaf22 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6625b94c8d1445e8afb60a6c5cfaf22" member_type: VOTER last_known_addr { host: "127.11.102.62" port: 41729 } } peers { permanent_uuid: "dcdaf0ddbf624baabcd4e95370ebb903" member_type: VOTER last_known_addr { host: "127.11.102.61" port: 34001 } } peers { permanent_uuid: "3cb87a0908e94d94aba2daae2c347e49" member_type: VOTER last_known_addr { host: "127.11.102.60" port: 35001 } } }
I20260620 06:09:19.535496 25250 sys_catalog.cc:458] T 00000000000000000000000000000000 P a6625b94c8d1445e8afb60a6c5cfaf22 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 06:09:19.536339 25264 catalog_manager.cc:1484] Initializing Kudu cluster ID...
W20260620 06:09:19.538481 25261 catalog_manager.cc:1558] T 00000000000000000000000000000000 P a6625b94c8d1445e8afb60a6c5cfaf22: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 06:09:19.538558 25261 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 06:09:19.539988 25211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:09:19.512423 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56728 (local address 127.11.102.61:34001)
0620 06:09:19.512538 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0620 06:09:19.512544 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 06:09:19.530900 (+ 18356us) server_negotiation.cc:407] Connection header received
0620 06:09:19.531082 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:09:19.531085 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:09:19.531133 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:09:19.531184 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:09:19.532046 (+ 862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:09:19.532839 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:09:19.533950 (+ 1111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:09:19.534343 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:09:19.534424 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:09:19.536400 (+ 1976us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:09:19.536404 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:09:19.536407 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:09:19.536436 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:09:19.538125 (+ 1689us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:09:19.538603 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:09:19.538606 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:09:19.538608 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:09:19.538710 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:09:19.538996 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:09:19.538999 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:09:19.539002 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:09:19.539232 (+ 230us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:09:19.539348 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0620 06:09:19.539697 (+ 349us) server_negotiation.cc:299] Negotiation successful
0620 06:09:19.539826 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260620 06:09:19.544185 25213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:09:19.512622 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39728 (local address 127.11.102.60:35001)
0620 06:09:19.514179 (+ 1557us) server_negotiation.cc:206] Beginning negotiation
0620 06:09:19.514183 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 06:09:19.531090 (+ 16907us) server_negotiation.cc:407] Connection header received
0620 06:09:19.531265 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:09:19.531268 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:09:19.531315 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:09:19.531365 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:09:19.532358 (+ 993us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:09:19.533124 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:09:19.534364 (+ 1240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:09:19.534699 (+ 335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:09:19.534777 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:09:19.540298 (+ 5521us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:09:19.540302 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:09:19.540305 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:09:19.540329 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:09:19.542146 (+ 1817us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:09:19.542762 (+ 616us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:09:19.542766 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:09:19.542768 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:09:19.542859 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:09:19.543156 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:09:19.543159 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:09:19.543161 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:09:19.543373 (+ 212us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:09:19.543531 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0620 06:09:19.543786 (+ 255us) server_negotiation.cc:299] Negotiation successful
0620 06:09:19.543909 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1482}
I20260620 06:09:19.556545 25029 raft_consensus.cc:1261] T 00000000000000000000000000000000 P a6625b94c8d1445e8afb60a6c5cfaf22 [term 1 FOLLOWER]: Refusing update from remote peer 3cb87a0908e94d94aba2daae2c347e49: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 06:09:19.556634 25104 raft_consensus.cc:1261] T 00000000000000000000000000000000 P dcdaf0ddbf624baabcd4e95370ebb903 [term 1 FOLLOWER]: Refusing update from remote peer 3cb87a0908e94d94aba2daae2c347e49: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 06:09:19.557399 25222 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 3cb87a0908e94d94aba2daae2c347e49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a6625b94c8d1445e8afb60a6c5cfaf22" member_type: VOTER last_known_addr { host: "127.11.102.62" port: 41729 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 06:09:19.557881 25222 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 3cb87a0908e94d94aba2daae2c347e49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dcdaf0ddbf624baabcd4e95370ebb903" member_type: VOTER last_known_addr { host: "127.11.102.61" port: 34001 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 06:09:19.562587 25250 sys_catalog.cc:455] T 00000000000000000000000000000000 P a6625b94c8d1445e8afb60a6c5cfaf22 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3cb87a0908e94d94aba2daae2c347e49. Latest consensus state: current_term: 1 leader_uuid: "3cb87a0908e94d94aba2daae2c347e49" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6625b94c8d1445e8afb60a6c5cfaf22" member_type: VOTER last_known_addr { host: "127.11.102.62" port: 41729 } } peers { permanent_uuid: "dcdaf0ddbf624baabcd4e95370ebb903" member_type: VOTER last_known_addr { host: "127.11.102.61" port: 34001 } } peers { permanent_uuid: "3cb87a0908e94d94aba2daae2c347e49" member_type: VOTER last_known_addr { host: "127.11.102.60" port: 35001 } } }
I20260620 06:09:19.562692 25250 sys_catalog.cc:458] T 00000000000000000000000000000000 P a6625b94c8d1445e8afb60a6c5cfaf22 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 06:09:19.567030 25221 sys_catalog.cc:455] T 00000000000000000000000000000000 P dcdaf0ddbf624baabcd4e95370ebb903 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3cb87a0908e94d94aba2daae2c347e49. Latest consensus state: current_term: 1 leader_uuid: "3cb87a0908e94d94aba2daae2c347e49" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6625b94c8d1445e8afb60a6c5cfaf22" member_type: VOTER last_known_addr { host: "127.11.102.62" port: 41729 } } peers { permanent_uuid: "dcdaf0ddbf624baabcd4e95370ebb903" member_type: VOTER last_known_addr { host: "127.11.102.61" port: 34001 } } peers { permanent_uuid: "3cb87a0908e94d94aba2daae2c347e49" member_type: VOTER last_known_addr { host: "127.11.102.60" port: 35001 } } }
I20260620 06:09:19.567178 25221 sys_catalog.cc:458] T 00000000000000000000000000000000 P dcdaf0ddbf624baabcd4e95370ebb903 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 06:09:19.572657 25262 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3cb87a0908e94d94aba2daae2c347e49 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "3cb87a0908e94d94aba2daae2c347e49" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6625b94c8d1445e8afb60a6c5cfaf22" member_type: VOTER last_known_addr { host: "127.11.102.62" port: 41729 } } peers { permanent_uuid: "dcdaf0ddbf624baabcd4e95370ebb903" member_type: VOTER last_known_addr { host: "127.11.102.61" port: 34001 } } peers { permanent_uuid: "3cb87a0908e94d94aba2daae2c347e49" member_type: VOTER last_known_addr { host: "127.11.102.60" port: 35001 } } }
I20260620 06:09:19.572782 25262 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3cb87a0908e94d94aba2daae2c347e49 [sys.catalog]: This master's current role is: LEADER
I20260620 06:09:19.572903 25266 mvcc.cc:205] Tried to move back new op lower bound from 7298808871137107968 to 7298808871034859520. Current Snapshot: MvccSnapshot[applied={T|T < 7298808871137107968}]
I20260620 06:09:19.573997 25250 sys_catalog.cc:455] T 00000000000000000000000000000000 P a6625b94c8d1445e8afb60a6c5cfaf22 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "3cb87a0908e94d94aba2daae2c347e49" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6625b94c8d1445e8afb60a6c5cfaf22" member_type: VOTER last_known_addr { host: "127.11.102.62" port: 41729 } } peers { permanent_uuid: "dcdaf0ddbf624baabcd4e95370ebb903" member_type: VOTER last_known_addr { host: "127.11.102.61" port: 34001 } } peers { permanent_uuid: "3cb87a0908e94d94aba2daae2c347e49" member_type: VOTER last_known_addr { host: "127.11.102.60" port: 35001 } } }
I20260620 06:09:19.574103 25250 sys_catalog.cc:458] T 00000000000000000000000000000000 P a6625b94c8d1445e8afb60a6c5cfaf22 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 06:09:19.574791 25267 mvcc.cc:205] Tried to move back new op lower bound from 7298808871137107968 to 7298808871034859520. Current Snapshot: MvccSnapshot[applied={T|T < 7298808871137107968}]
I20260620 06:09:19.577905 25222 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3cb87a0908e94d94aba2daae2c347e49 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "3cb87a0908e94d94aba2daae2c347e49" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6625b94c8d1445e8afb60a6c5cfaf22" member_type: VOTER last_known_addr { host: "127.11.102.62" port: 41729 } } peers { permanent_uuid: "dcdaf0ddbf624baabcd4e95370ebb903" member_type: VOTER last_known_addr { host: "127.11.102.61" port: 34001 } } peers { permanent_uuid: "3cb87a0908e94d94aba2daae2c347e49" member_type: VOTER last_known_addr { host: "127.11.102.60" port: 35001 } } }
I20260620 06:09:19.578023 25222 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3cb87a0908e94d94aba2daae2c347e49 [sys.catalog]: This master's current role is: LEADER
I20260620 06:09:19.578853 25269 mvcc.cc:205] Tried to move back new op lower bound from 7298808871137107968 to 7298808871034859520. Current Snapshot: MvccSnapshot[applied={T|T < 7298808871137107968}]
I20260620 06:09:19.578976 25221 sys_catalog.cc:455] T 00000000000000000000000000000000 P dcdaf0ddbf624baabcd4e95370ebb903 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "3cb87a0908e94d94aba2daae2c347e49" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6625b94c8d1445e8afb60a6c5cfaf22" member_type: VOTER last_known_addr { host: "127.11.102.62" port: 41729 } } peers { permanent_uuid: "dcdaf0ddbf624baabcd4e95370ebb903" member_type: VOTER last_known_addr { host: "127.11.102.61" port: 34001 } } peers { permanent_uuid: "3cb87a0908e94d94aba2daae2c347e49" member_type: VOTER last_known_addr { host: "127.11.102.60" port: 35001 } } }
I20260620 06:09:19.579077 25221 sys_catalog.cc:458] T 00000000000000000000000000000000 P dcdaf0ddbf624baabcd4e95370ebb903 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 06:09:19.580906 25264 catalog_manager.cc:1347] Generated new cluster ID: 12721f84763442c8ab38aa40fadd02b6
I20260620 06:09:19.580971 25264 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 06:09:19.601514 25264 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 06:09:19.602738 25264 catalog_manager.cc:1504] Loading token signing keys...
I20260620 06:09:19.626055 25264 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3cb87a0908e94d94aba2daae2c347e49: Generated new TSK 0
I20260620 06:09:19.627003 25264 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 06:09:20.519721 25247 catalog_manager.cc:1259] Loaded cluster ID: 12721f84763442c8ab38aa40fadd02b6
I20260620 06:09:20.519827 25247 catalog_manager.cc:1552] T 00000000000000000000000000000000 P dcdaf0ddbf624baabcd4e95370ebb903: loading cluster ID for follower catalog manager: success
I20260620 06:09:20.521499 25247 catalog_manager.cc:1574] T 00000000000000000000000000000000 P dcdaf0ddbf624baabcd4e95370ebb903: acquiring CA information for follower catalog manager: success
I20260620 06:09:20.522785 25247 catalog_manager.cc:1602] T 00000000000000000000000000000000 P dcdaf0ddbf624baabcd4e95370ebb903: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 06:09:20.539708 25261 catalog_manager.cc:1259] Loaded cluster ID: 12721f84763442c8ab38aa40fadd02b6
I20260620 06:09:20.539777 25261 catalog_manager.cc:1552] T 00000000000000000000000000000000 P a6625b94c8d1445e8afb60a6c5cfaf22: loading cluster ID for follower catalog manager: success
I20260620 06:09:20.541460 25261 catalog_manager.cc:1574] T 00000000000000000000000000000000 P a6625b94c8d1445e8afb60a6c5cfaf22: acquiring CA information for follower catalog manager: success
I20260620 06:09:20.542577 25261 catalog_manager.cc:1602] T 00000000000000000000000000000000 P a6625b94c8d1445e8afb60a6c5cfaf22: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 06:09:20.711052 11672 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9HMp43/build/debug/bin/kudu with pid 24987
I20260620 06:09:20.717717 11672 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9HMp43/build/debug/bin/kudu with pid 25062
I20260620 06:09:20.723541 11672 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9HMp43/build/debug/bin/kudu with pid 25136
2026-06-20T06:09:20Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (3015 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781935603120653-11672-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 06:09:20 dist-test-slave-9hm5 krb5kdc[25281](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 06:09:20 dist-test-slave-9hm5 krb5kdc[25281](info): set up 2 sockets
Jun 20 06:09:20 dist-test-slave-9hm5 krb5kdc[25281](info): commencing operation
krb5kdc: starting...
W20260620 06:09:21.759982 11672 mini_kdc.cc:121] Time spent starting KDC: real 1.015s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781935603120653-11672-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781935603120653-11672-0/krb5kdc/test-user.keytab.
Jun 20 06:09:21 dist-test-slave-9hm5 krb5kdc[25281](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935761, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T06:09:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T06:09:21Z Disabled control of system clock
WARNING: no policy specified for kudu/127.11.102.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.102.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.102.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781935603120653-11672-0/krb5kdc/kudu_127.11.102.62.keytab.
Entry for principal kudu/127.11.102.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781935603120653-11672-0/krb5kdc/kudu_127.11.102.62.keytab.
I20260620 06:09:21.882201 11672 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9HMp43/build/debug/bin/kudu
/tmp/dist-test-task9HMp43/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781935603120653-11672-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781935603120653-11672-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781935603120653-11672-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781935603120653-11672-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.102.62:32969
--webserver_interface=127.11.102.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781935603120653-11672-0/krb5kdc/kudu_127.11.102.62.keytab
--principal=kudu/127.11.102.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.102.20:41091
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.102.62:32969
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781935603120653-11672-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781935603120653-11672-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781935603120653-11672-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 06:09:21.991815 25294 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:09:21.992066 25294 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:09:21.992106 25294 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:09:21.995954 25294 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 06:09:21.995996 25294 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:09:21.996016 25294 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:09:21.996031 25294 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 06:09:21.996047 25294 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 06:09:22.001199 25294 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.102.20:41091
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781935603120653-11672-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781935603120653-11672-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.102.62:32969
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781935603120653-11672-0/krb5kdc/kudu_127.11.102.62.keytab
--principal=kudu/127.11.102.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.102.62:32969
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781935603120653-11672-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.102.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25294
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781935603120653-11672-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 05:28:48 UTC on ccycloud.quasar-fixhms.root.comops.site
build id 4457020
I20260620 06:09:22.002136 25294 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:09:22.003075 25294 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 06:09:22.012646 25300 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:09:22.012632 25299 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:09:22.012624 25302 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:09:22.012678 25294 server_base.cc:1034] running on GCE node
I20260620 06:09:22.013528 25294 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:09:22.014636 25294 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:09:22.015823 25294 hybrid_clock.cc:648] HybridClock initialized: now 1781935762015815 us; error 29 us; skew 500 ppm
Jun 20 06:09:22 dist-test-slave-9hm5 krb5kdc[25281](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935762, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.102.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 06:09:22.018613 25294 init.cc:377] Logged in from keytab as kudu/127.11.102.62@KRBTEST.COM (short username kudu)
I20260620 06:09:22.019505 25294 webserver.cc:460] Webserver started at http://127.11.102.62:32779/ using document root <none> and password file <none>
I20260620 06:09:22.020128 25294 fs_manager.cc:359] Metadata directory not provided
I20260620 06:09:22.020234 25294 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:09:22.020475 25294 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 06:09:22.022578 25294 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781935603120653-11672-0/minicluster-data/master-0/data/instance:
uuid: "a0a8ed56025641fb974fc92cfe725cd3"
format_stamp: "Formatted at 2026-06-20 06:09:22 on dist-test-slave-9hm5"
server_key: "501d5180bd7924db90e5c8463f2c01da"
server_key_iv: "a86f709173f2cba5e8e14234af20e54f"
server_key_version: "encryptionkey@0"
I20260620 06:09:22.023156 25294 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781935603120653-11672-0/minicluster-data/master-0/wal/instance:
uuid: "a0a8ed56025641fb974fc92cfe725cd3"
format_stamp: "Formatted at 2026-06-20 06:09:22 on dist-test-slave-9hm5"
server_key: "501d5180bd7924db90e5c8463f2c01da"
server_key_iv: "a86f709173f2cba5e8e14234af20e54f"
server_key_version: "encryptionkey@0"
I20260620 06:09:22.026861 25294 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260620 06:09:22.029289 25309 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:09:22.030495 25294 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260620 06:09:22.030635 25294 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781935603120653-11672-0/minicluster-data/master-0/data,/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781935603120653-11672-0/minicluster-data/master-0/wal
uuid: "a0a8ed56025641fb974fc92cfe725cd3"
format_stamp: "Formatted at 2026-06-20 06:09:22 on dist-test-slave-9hm5"
server_key: "501d5180bd7924db90e5c8463f2c01da"
server_key_iv: "a86f709173f2cba5e8e14234af20e54f"
server_key_version: "encryptionkey@0"
I20260620 06:09:22.030747 25294 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781935603120653-11672-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781935603120653-11672-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781935603120653-11672-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:09:22.047922 25294 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:09:22.050843 25294 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:09:22.051059 25294 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:09:22.058131 25294 rpc_server.cc:307] RPC server started. Bound to: 127.11.102.62:32969
I20260620 06:09:22.058159 25361 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.102.62:32969 every 8 connection(s)
I20260620 06:09:22.059073 25294 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781935603120653-11672-0/minicluster-data/master-0/data/info.pb
I20260620 06:09:22.061800 25362 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 06:09:22.067385 25362 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a0a8ed56025641fb974fc92cfe725cd3: Bootstrap starting.
I20260620 06:09:22.068815 11672 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9HMp43/build/debug/bin/kudu as pid 25294
I20260620 06:09:22.068941 11672 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781935603120653-11672-0/minicluster-data/master-0/wal/instance
I20260620 06:09:22.069223 11672 external_mini_cluster.cc:1428] Setting key 7a377baa97530ef1bacfe26c15062bf0
I20260620 06:09:22.070164 25362 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a0a8ed56025641fb974fc92cfe725cd3: Neither blocks nor log segments found. Creating new log.
I20260620 06:09:22.070979 25362 log.cc:824] T 00000000000000000000000000000000 P a0a8ed56025641fb974fc92cfe725cd3: Log is configured to *not* fsync() on all Append() calls
I20260620 06:09:22.072863 25362 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a0a8ed56025641fb974fc92cfe725cd3: No bootstrap required, opened a new log
I20260620 06:09:22.076285 25362 raft_consensus.cc:348] T 00000000000000000000000000000000 P a0a8ed56025641fb974fc92cfe725cd3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a0a8ed56025641fb974fc92cfe725cd3" member_type: VOTER last_known_addr { host: "127.11.102.62" port: 32969 } }
I20260620 06:09:22.076519 25362 raft_consensus.cc:374] T 00000000000000000000000000000000 P a0a8ed56025641fb974fc92cfe725cd3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 06:09:22.076604 25362 raft_consensus.cc:729] T 00000000000000000000000000000000 P a0a8ed56025641fb974fc92cfe725cd3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a0a8ed56025641fb974fc92cfe725cd3, State: Initialized, Role: FOLLOWER
I20260620 06:09:22.077090 25362 consensus_queue.cc:260] T 00000000000000000000000000000000 P a0a8ed56025641fb974fc92cfe725cd3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a0a8ed56025641fb974fc92cfe725cd3" member_type: VOTER last_known_addr { host: "127.11.102.62" port: 32969 } }
I20260620 06:09:22.077255 25362 raft_consensus.cc:388] T 00000000000000000000000000000000 P a0a8ed56025641fb974fc92cfe725cd3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 06:09:22.077322 25362 raft_consensus.cc:482] T 00000000000000000000000000000000 P a0a8ed56025641fb974fc92cfe725cd3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 06:09:22.077418 25362 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a0a8ed56025641fb974fc92cfe725cd3 [term 0 FOLLOWER]: Advancing to term 1
Jun 20 06:09:22 dist-test-slave-9hm5 krb5kdc[25281](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935761, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.102.62@KRBTEST.COM
I20260620 06:09:22.078399 25362 raft_consensus.cc:504] T 00000000000000000000000000000000 P a0a8ed56025641fb974fc92cfe725cd3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a0a8ed56025641fb974fc92cfe725cd3" member_type: VOTER last_known_addr { host: "127.11.102.62" port: 32969 } }
I20260620 06:09:22.078761 25362 leader_election.cc:302] T 00000000000000000000000000000000 P a0a8ed56025641fb974fc92cfe725cd3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a0a8ed56025641fb974fc92cfe725cd3; no voters:
I20260620 06:09:22.079079 25362 leader_election.cc:288] T 00000000000000000000000000000000 P a0a8ed56025641fb974fc92cfe725cd3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 06:09:22.079418 25368 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a0a8ed56025641fb974fc92cfe725cd3 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 06:09:22.080427 25362 sys_catalog.cc:564] T 00000000000000000000000000000000 P a0a8ed56025641fb974fc92cfe725cd3 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 06:09:22.080857 25368 raft_consensus.cc:686] T 00000000000000000000000000000000 P a0a8ed56025641fb974fc92cfe725cd3 [term 1 LEADER]: Becoming Leader. State: Replica: a0a8ed56025641fb974fc92cfe725cd3, State: Running, Role: LEADER
I20260620 06:09:22.081192 25368 consensus_queue.cc:237] T 00000000000000000000000000000000 P a0a8ed56025641fb974fc92cfe725cd3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a0a8ed56025641fb974fc92cfe725cd3" member_type: VOTER last_known_addr { host: "127.11.102.62" port: 32969 } }
I20260620 06:09:22.082969 25367 sys_catalog.cc:455] T 00000000000000000000000000000000 P a0a8ed56025641fb974fc92cfe725cd3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a0a8ed56025641fb974fc92cfe725cd3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a0a8ed56025641fb974fc92cfe725cd3" member_type: VOTER last_known_addr { host: "127.11.102.62" port: 32969 } } }
I20260620 06:09:22.083652 25367 sys_catalog.cc:458] T 00000000000000000000000000000000 P a0a8ed56025641fb974fc92cfe725cd3 [sys.catalog]: This master's current role is: LEADER
I20260620 06:09:22.083220 25369 sys_catalog.cc:455] T 00000000000000000000000000000000 P a0a8ed56025641fb974fc92cfe725cd3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a0a8ed56025641fb974fc92cfe725cd3. Latest consensus state: current_term: 1 leader_uuid: "a0a8ed56025641fb974fc92cfe725cd3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a0a8ed56025641fb974fc92cfe725cd3" member_type: VOTER last_known_addr { host: "127.11.102.62" port: 32969 } } }
I20260620 06:09:22.083878 25369 sys_catalog.cc:458] T 00000000000000000000000000000000 P a0a8ed56025641fb974fc92cfe725cd3 [sys.catalog]: This master's current role is: LEADER
I20260620 06:09:22.084728 25365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:09:22.070340 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35304 (local address 127.11.102.62:32969)
0620 06:09:22.070786 (+ 446us) server_negotiation.cc:206] Beginning negotiation
0620 06:09:22.070804 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0620 06:09:22.070838 (+ 34us) server_negotiation.cc:407] Connection header received
0620 06:09:22.071528 (+ 690us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:09:22.071546 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:09:22.071948 (+ 402us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:09:22.072291 (+ 343us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:09:22.073061 (+ 770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:09:22.074037 (+ 976us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:09:22.074941 (+ 904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:09:22.075415 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:09:22.075594 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:09:22.078406 (+ 2812us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:09:22.078416 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:09:22.078428 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:09:22.078454 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:09:22.080634 (+ 2180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:09:22.081232 (+ 598us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:09:22.081239 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:09:22.081244 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:09:22.081400 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:09:22.081729 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:09:22.081733 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:09:22.081735 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:09:22.082262 (+ 527us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:09:22.082495 (+ 233us) server_negotiation.cc:1035] Waiting for connection context
0620 06:09:22.083522 (+ 1027us) server_negotiation.cc:299] Negotiation successful
0620 06:09:22.083877 (+ 355us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":307,"thread_start_us":131,"threads_started":1}
W20260620 06:09:22.085459 25381 catalog_manager.cc:1558] T 00000000000000000000000000000000 P a0a8ed56025641fb974fc92cfe725cd3: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 06:09:22.085553 25381 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 06:09:22.087678 25382 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 06:09:22.087692 11672 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 06:09:22.088639 25382 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 06:09:22.093655 25382 catalog_manager.cc:1347] Generated new cluster ID: 59ae3ff0e4c14d7ba372f64bed95865d
I20260620 06:09:22.093716 25382 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 06:09:22.107551 25382 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 06:09:22.108424 25382 catalog_manager.cc:1504] Loading token signing keys...
I20260620 06:09:22.115160 25382 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a0a8ed56025641fb974fc92cfe725cd3: Generated new TSK 0
I20260620 06:09:22.116248 25382 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 06:09:22.143421 11672 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9HMp43/build/debug/bin/kudu with pid 25294
2026-06-20T06:09:22Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (1418 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935603120653-11672-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 06:09:22 dist-test-slave-9hm5 krb5kdc[25389](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 06:09:22 dist-test-slave-9hm5 krb5kdc[25389](info): set up 2 sockets
Jun 20 06:09:22 dist-test-slave-9hm5 krb5kdc[25389](info): commencing operation
krb5kdc: starting...
W20260620 06:09:23.184123 11672 mini_kdc.cc:121] Time spent starting KDC: real 1.021s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935603120653-11672-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935603120653-11672-0/krb5kdc/test-user.keytab.
Jun 20 06:09:23 dist-test-slave-9hm5 krb5kdc[25389](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935763, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T06:09:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T06:09:23Z Disabled control of system clock
WARNING: no policy specified for kudu/127.11.102.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.102.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.102.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935603120653-11672-0/krb5kdc/kudu_127.11.102.62.keytab.
Entry for principal kudu/127.11.102.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935603120653-11672-0/krb5kdc/kudu_127.11.102.62.keytab.
I20260620 06:09:23.304883 11672 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9HMp43/build/debug/bin/kudu
/tmp/dist-test-task9HMp43/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935603120653-11672-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935603120653-11672-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935603120653-11672-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935603120653-11672-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.102.62:41475
--webserver_interface=127.11.102.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935603120653-11672-0/krb5kdc/kudu_127.11.102.62.keytab
--principal=kudu/127.11.102.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.102.20:37051
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.102.62:41475
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935603120653-11672-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935603120653-11672-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935603120653-11672-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 06:09:23.411826 25402 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:09:23.412070 25402 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:09:23.412108 25402 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:09:23.416066 25402 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 06:09:23.416118 25402 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:09:23.416138 25402 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:09:23.416154 25402 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 06:09:23.416169 25402 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 06:09:23.421527 25402 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.102.20:37051
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935603120653-11672-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935603120653-11672-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.102.62:41475
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935603120653-11672-0/krb5kdc/kudu_127.11.102.62.keytab
--principal=kudu/127.11.102.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.102.62:41475
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935603120653-11672-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.102.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25402
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935603120653-11672-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 05:28:48 UTC on ccycloud.quasar-fixhms.root.comops.site
build id 4457020
I20260620 06:09:23.422542 25402 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:09:23.423559 25402 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 06:09:23.432945 25407 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:09:23.432962 25410 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:09:23.432945 25408 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:09:23.433238 25402 server_base.cc:1034] running on GCE node
I20260620 06:09:23.433768 25402 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:09:23.434789 25402 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:09:23.436019 25402 hybrid_clock.cc:648] HybridClock initialized: now 1781935763435990 us; error 48 us; skew 500 ppm
Jun 20 06:09:23 dist-test-slave-9hm5 krb5kdc[25389](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935763, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.102.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 06:09:23.438680 25402 init.cc:377] Logged in from keytab as kudu/127.11.102.62@KRBTEST.COM (short username kudu)
I20260620 06:09:23.439576 25402 webserver.cc:460] Webserver started at http://127.11.102.62:43041/ using document root <none> and password file <none>
I20260620 06:09:23.440201 25402 fs_manager.cc:359] Metadata directory not provided
I20260620 06:09:23.440308 25402 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:09:23.440517 25402 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 06:09:23.442493 25402 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935603120653-11672-0/minicluster-data/master-0/data/instance:
uuid: "6743bd85c057436695ea713d0e6cd57b"
format_stamp: "Formatted at 2026-06-20 06:09:23 on dist-test-slave-9hm5"
server_key: "bc428878fb4faafd9bd236886d35abf7"
server_key_iv: "aa844420908e3443787e4f2fb6edd13e"
server_key_version: "encryptionkey@0"
I20260620 06:09:23.443053 25402 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935603120653-11672-0/minicluster-data/master-0/wal/instance:
uuid: "6743bd85c057436695ea713d0e6cd57b"
format_stamp: "Formatted at 2026-06-20 06:09:23 on dist-test-slave-9hm5"
server_key: "bc428878fb4faafd9bd236886d35abf7"
server_key_iv: "aa844420908e3443787e4f2fb6edd13e"
server_key_version: "encryptionkey@0"
I20260620 06:09:23.446605 25402 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.003s
I20260620 06:09:23.449055 25417 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:09:23.450183 25402 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260620 06:09:23.450354 25402 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935603120653-11672-0/minicluster-data/master-0/data,/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935603120653-11672-0/minicluster-data/master-0/wal
uuid: "6743bd85c057436695ea713d0e6cd57b"
format_stamp: "Formatted at 2026-06-20 06:09:23 on dist-test-slave-9hm5"
server_key: "bc428878fb4faafd9bd236886d35abf7"
server_key_iv: "aa844420908e3443787e4f2fb6edd13e"
server_key_version: "encryptionkey@0"
I20260620 06:09:23.450439 25402 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935603120653-11672-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935603120653-11672-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935603120653-11672-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:09:23.478314 25402 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:09:23.481047 25402 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:09:23.481194 25402 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:09:23.487920 25402 rpc_server.cc:307] RPC server started. Bound to: 127.11.102.62:41475
I20260620 06:09:23.488206 25469 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.102.62:41475 every 8 connection(s)
I20260620 06:09:23.488866 25402 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935603120653-11672-0/minicluster-data/master-0/data/info.pb
I20260620 06:09:23.491019 11672 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9HMp43/build/debug/bin/kudu as pid 25402
I20260620 06:09:23.491166 11672 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935603120653-11672-0/minicluster-data/master-0/wal/instance
I20260620 06:09:23.491461 11672 external_mini_cluster.cc:1428] Setting key 9668a252d16580d7b1f81ca2471f81dd
I20260620 06:09:23.491701 25470 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 06:09:23.498618 25470 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6743bd85c057436695ea713d0e6cd57b: Bootstrap starting.
Jun 20 06:09:23 dist-test-slave-9hm5 krb5kdc[25389](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935763, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.102.62@KRBTEST.COM
I20260620 06:09:23.500993 25470 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6743bd85c057436695ea713d0e6cd57b: Neither blocks nor log segments found. Creating new log.
I20260620 06:09:23.501761 25470 log.cc:824] T 00000000000000000000000000000000 P 6743bd85c057436695ea713d0e6cd57b: Log is configured to *not* fsync() on all Append() calls
I20260620 06:09:23.503589 25470 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6743bd85c057436695ea713d0e6cd57b: No bootstrap required, opened a new log
I20260620 06:09:23.507580 25473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:09:23.492905 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48644 (local address 127.11.102.62:41475)
0620 06:09:23.493326 (+ 421us) server_negotiation.cc:206] Beginning negotiation
0620 06:09:23.493344 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0620 06:09:23.493384 (+ 40us) server_negotiation.cc:407] Connection header received
0620 06:09:23.494344 (+ 960us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:09:23.494368 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:09:23.494753 (+ 385us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:09:23.495117 (+ 364us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:09:23.495818 (+ 701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:09:23.497254 (+ 1436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:09:23.498097 (+ 843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:09:23.498779 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:09:23.498924 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:09:23.501527 (+ 2603us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:09:23.501532 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:09:23.501547 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:09:23.501579 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:09:23.504589 (+ 3010us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:09:23.504991 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:09:23.504996 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:09:23.505002 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:09:23.505128 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:09:23.505379 (+ 251us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:09:23.505382 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:09:23.505384 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:09:23.505951 (+ 567us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:09:23.506153 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0620 06:09:23.506400 (+ 247us) server_negotiation.cc:299] Negotiation successful
0620 06:09:23.506673 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":262,"thread_start_us":109,"threads_started":1}
I20260620 06:09:23.508031 25470 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6743bd85c057436695ea713d0e6cd57b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6743bd85c057436695ea713d0e6cd57b" member_type: VOTER last_known_addr { host: "127.11.102.62" port: 41475 } }
I20260620 06:09:23.508261 25470 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6743bd85c057436695ea713d0e6cd57b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 06:09:23.508349 25470 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6743bd85c057436695ea713d0e6cd57b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6743bd85c057436695ea713d0e6cd57b, State: Initialized, Role: FOLLOWER
I20260620 06:09:23.508806 25470 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6743bd85c057436695ea713d0e6cd57b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6743bd85c057436695ea713d0e6cd57b" member_type: VOTER last_known_addr { host: "127.11.102.62" port: 41475 } }
I20260620 06:09:23.508956 25470 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6743bd85c057436695ea713d0e6cd57b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 06:09:23.509020 25470 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6743bd85c057436695ea713d0e6cd57b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 06:09:23.509110 25470 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6743bd85c057436695ea713d0e6cd57b [term 0 FOLLOWER]: Advancing to term 1
I20260620 06:09:23.510078 25470 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6743bd85c057436695ea713d0e6cd57b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6743bd85c057436695ea713d0e6cd57b" member_type: VOTER last_known_addr { host: "127.11.102.62" port: 41475 } }
I20260620 06:09:23.510361 25470 leader_election.cc:302] T 00000000000000000000000000000000 P 6743bd85c057436695ea713d0e6cd57b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6743bd85c057436695ea713d0e6cd57b; no voters:
I20260620 06:09:23.510586 25470 leader_election.cc:288] T 00000000000000000000000000000000 P 6743bd85c057436695ea713d0e6cd57b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 06:09:23.510675 25475 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6743bd85c057436695ea713d0e6cd57b [term 1 FOLLOWER]: Leader election won for term 1
I20260620 06:09:23.510932 25475 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6743bd85c057436695ea713d0e6cd57b [term 1 LEADER]: Becoming Leader. State: Replica: 6743bd85c057436695ea713d0e6cd57b, State: Running, Role: LEADER
I20260620 06:09:23.511256 25475 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6743bd85c057436695ea713d0e6cd57b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6743bd85c057436695ea713d0e6cd57b" member_type: VOTER last_known_addr { host: "127.11.102.62" port: 41475 } }
I20260620 06:09:23.511837 25470 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6743bd85c057436695ea713d0e6cd57b [sys.catalog]: configured and running, proceeding with master startup.
I20260620 06:09:23.512897 25477 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6743bd85c057436695ea713d0e6cd57b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6743bd85c057436695ea713d0e6cd57b. Latest consensus state: current_term: 1 leader_uuid: "6743bd85c057436695ea713d0e6cd57b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6743bd85c057436695ea713d0e6cd57b" member_type: VOTER last_known_addr { host: "127.11.102.62" port: 41475 } } }
I20260620 06:09:23.513073 25477 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6743bd85c057436695ea713d0e6cd57b [sys.catalog]: This master's current role is: LEADER
I20260620 06:09:23.513163 25476 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6743bd85c057436695ea713d0e6cd57b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6743bd85c057436695ea713d0e6cd57b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6743bd85c057436695ea713d0e6cd57b" member_type: VOTER last_known_addr { host: "127.11.102.62" port: 41475 } } }
I20260620 06:09:23.513301 25476 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6743bd85c057436695ea713d0e6cd57b [sys.catalog]: This master's current role is: LEADER
I20260620 06:09:23.513574 25484 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 06:09:23.516624 25484 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 06:09:23.521732 25484 catalog_manager.cc:1347] Generated new cluster ID: c8554031cfe44467b49feb636b18c097
I20260620 06:09:23.521795 25484 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 06:09:23.541102 25484 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 06:09:23.542089 25484 catalog_manager.cc:1504] Loading token signing keys...
I20260620 06:09:23.547647 25484 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6743bd85c057436695ea713d0e6cd57b: Generated new TSK 0
I20260620 06:09:23.548313 25484 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.11.102.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.102.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.102.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935603120653-11672-0/krb5kdc/kudu_127.11.102.1.keytab.
Entry for principal kudu/127.11.102.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935603120653-11672-0/krb5kdc/kudu_127.11.102.1.keytab.
I20260620 06:09:23.579012 11672 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9HMp43/build/debug/bin/kudu
/tmp/dist-test-task9HMp43/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935603120653-11672-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935603120653-11672-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935603120653-11672-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935603120653-11672-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.102.1:0
--local_ip_for_outbound_sockets=127.11.102.1
--webserver_interface=127.11.102.1
--webserver_port=0
--tserver_master_addrs=127.11.102.62:41475
--keytab_file=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935603120653-11672-0/krb5kdc/kudu_127.11.102.1.keytab
--principal=kudu/127.11.102.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.102.20:37051
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935603120653-11672-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935603120653-11672-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935603120653-11672-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 06:09:23.690233 25496 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:09:23.690478 25496 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:09:23.690518 25496 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:09:23.694487 25496 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:09:23.694535 25496 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:09:23.694621 25496 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.102.1
I20260620 06:09:23.699734 25496 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.102.20:37051
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935603120653-11672-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935603120653-11672-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935603120653-11672-0/krb5kdc/kudu_127.11.102.1.keytab
--principal=kudu/127.11.102.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.102.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935603120653-11672-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.11.102.1
--webserver_port=0
--tserver_master_addrs=127.11.102.62:41475
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25496
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.102.1
--log_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935603120653-11672-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 05:28:48 UTC on ccycloud.quasar-fixhms.root.comops.site
build id 4457020
I20260620 06:09:23.700661 25496 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:09:23.701632 25496 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 06:09:23.710945 25501 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:09:23.711035 25496 server_base.cc:1034] running on GCE node
W20260620 06:09:23.710963 25504 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:09:23.710945 25502 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:09:23.711557 25496 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:09:23.712270 25496 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:09:23.713424 25496 hybrid_clock.cc:648] HybridClock initialized: now 1781935763713411 us; error 25 us; skew 500 ppm
Jun 20 06:09:23 dist-test-slave-9hm5 krb5kdc[25389](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935763, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.102.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 06:09:23.716068 25496 init.cc:377] Logged in from keytab as kudu/127.11.102.1@KRBTEST.COM (short username kudu)
I20260620 06:09:23.716799 25496 webserver.cc:460] Webserver started at http://127.11.102.1:42581/ using document root <none> and password file <none>
I20260620 06:09:23.717275 25496 fs_manager.cc:359] Metadata directory not provided
I20260620 06:09:23.717338 25496 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:09:23.717489 25496 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 06:09:23.719318 25496 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935603120653-11672-0/minicluster-data/ts-0/data/instance:
uuid: "d7f871fdb6814adabd0538690e2a65a3"
format_stamp: "Formatted at 2026-06-20 06:09:23 on dist-test-slave-9hm5"
server_key: "110aac0377228e58ef7c06a5891315b6"
server_key_iv: "389c4dcb9928faf9e8cd6baa4e0b02f5"
server_key_version: "encryptionkey@0"
I20260620 06:09:23.719874 25496 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935603120653-11672-0/minicluster-data/ts-0/wal/instance:
uuid: "d7f871fdb6814adabd0538690e2a65a3"
format_stamp: "Formatted at 2026-06-20 06:09:23 on dist-test-slave-9hm5"
server_key: "110aac0377228e58ef7c06a5891315b6"
server_key_iv: "389c4dcb9928faf9e8cd6baa4e0b02f5"
server_key_version: "encryptionkey@0"
I20260620 06:09:23.723291 25496 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.002s
I20260620 06:09:23.725852 25511 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:09:23.727146 25496 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 06:09:23.727308 25496 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935603120653-11672-0/minicluster-data/ts-0/data,/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935603120653-11672-0/minicluster-data/ts-0/wal
uuid: "d7f871fdb6814adabd0538690e2a65a3"
format_stamp: "Formatted at 2026-06-20 06:09:23 on dist-test-slave-9hm5"
server_key: "110aac0377228e58ef7c06a5891315b6"
server_key_iv: "389c4dcb9928faf9e8cd6baa4e0b02f5"
server_key_version: "encryptionkey@0"
I20260620 06:09:23.727424 25496 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935603120653-11672-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935603120653-11672-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935603120653-11672-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:09:23.750638 25496 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:09:23.753681 25496 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:09:23.753899 25496 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:09:23.754594 25496 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 06:09:23.755545 25496 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 06:09:23.755594 25496 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 06:09:23.755694 25496 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 06:09:23.755723 25496 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 06:09:23.765230 25496 rpc_server.cc:307] RPC server started. Bound to: 127.11.102.1:41679
I20260620 06:09:23.765246 25624 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.102.1:41679 every 8 connection(s)
I20260620 06:09:23.766492 25496 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935603120653-11672-0/minicluster-data/ts-0/data/info.pb
I20260620 06:09:23.775285 11672 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9HMp43/build/debug/bin/kudu as pid 25496
I20260620 06:09:23.775434 11672 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935603120653-11672-0/minicluster-data/ts-0/wal/instance
I20260620 06:09:23.775801 11672 external_mini_cluster.cc:1428] Setting key 3b2086295d08a472c5562c8fa3393f9c
Jun 20 06:09:23 dist-test-slave-9hm5 krb5kdc[25389](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935763, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.102.1@KRBTEST.COM for kudu/127.11.102.62@KRBTEST.COM
I20260620 06:09:23.780915 25473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:09:23.769125 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.102.1:57887 (local address 127.11.102.62:41475)
0620 06:09:23.769289 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0620 06:09:23.769295 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 06:09:23.769985 (+ 690us) server_negotiation.cc:407] Connection header received
0620 06:09:23.771483 (+ 1498us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:09:23.771487 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:09:23.771534 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:09:23.771667 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:09:23.772758 (+ 1091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:09:23.773278 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:09:23.774352 (+ 1074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:09:23.774577 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:09:23.774654 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:09:23.777280 (+ 2626us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:09:23.777284 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:09:23.777286 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:09:23.777314 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:09:23.778778 (+ 1464us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:09:23.779307 (+ 529us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:09:23.779312 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:09:23.779314 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:09:23.779396 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:09:23.779829 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:09:23.779834 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:09:23.779836 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:09:23.780013 (+ 177us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:09:23.780130 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0620 06:09:23.780622 (+ 492us) server_negotiation.cc:299] Negotiation successful
0620 06:09:23.780746 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260620 06:09:23.781733 25627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:09:23.769227 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.102.62:41475 (local address 127.11.102.1:57887)
0620 06:09:23.769799 (+ 572us) negotiation.cc:107] Waiting for socket to connect
0620 06:09:23.769862 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0620 06:09:23.771277 (+ 1415us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:09:23.771813 (+ 536us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:09:23.771820 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:09:23.772171 (+ 351us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:09:23.772623 (+ 452us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:09:23.772635 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:09:23.773389 (+ 754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:09:23.773391 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:09:23.774247 (+ 856us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:09:23.774256 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:09:23.774704 (+ 448us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:09:23.774707 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:09:23.774846 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:09:23.775530 (+ 684us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:09:23.775549 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:09:23.777113 (+ 1564us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:09:23.778914 (+ 1801us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:09:23.778919 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:09:23.778933 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:09:23.779191 (+ 258us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:09:23.779502 (+ 311us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:09:23.779505 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:09:23.779507 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:09:23.779726 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:09:23.780126 (+ 400us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:09:23.780132 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:09:23.780378 (+ 246us) client_negotiation.cc:769] Sending connection context
0620 06:09:23.780566 (+ 188us) client_negotiation.cc:240] Negotiation successful
0620 06:09:23.780886 (+ 320us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":334,"thread_start_us":127,"threads_started":1}
I20260620 06:09:23.782994 25625 heartbeater.cc:347] Connected to a master server at 127.11.102.62:41475
I20260620 06:09:23.783257 25625 heartbeater.cc:464] Registering TS with master...
I20260620 06:09:23.783849 25625 heartbeater.cc:511] Master 127.11.102.62:41475 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.11.102.2@KRBTEST.COM; defaulting to no policy
I20260620 06:09:23.785518 25434 ts_manager.cc:194] Registered new tserver with Master: d7f871fdb6814adabd0538690e2a65a3 (127.11.102.1:41679)
I20260620 06:09:23.786618 25434 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.102.1@KRBTEST.COM'} at 127.11.102.1:57887
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.102.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.102.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935603120653-11672-0/krb5kdc/kudu_127.11.102.2.keytab.
Entry for principal kudu/127.11.102.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935603120653-11672-0/krb5kdc/kudu_127.11.102.2.keytab.
I20260620 06:09:23.801956 11672 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9HMp43/build/debug/bin/kudu
/tmp/dist-test-task9HMp43/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935603120653-11672-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935603120653-11672-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935603120653-11672-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935603120653-11672-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.102.2:0
--local_ip_for_outbound_sockets=127.11.102.2
--webserver_interface=127.11.102.2
--webserver_port=0
--tserver_master_addrs=127.11.102.62:41475
--keytab_file=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935603120653-11672-0/krb5kdc/kudu_127.11.102.2.keytab
--principal=kudu/127.11.102.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.102.20:37051
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935603120653-11672-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935603120653-11672-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935603120653-11672-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 06:09:23.909643 25630 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:09:23.909879 25630 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:09:23.909920 25630 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:09:23.914103 25630 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:09:23.914157 25630 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:09:23.914217 25630 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.102.2
I20260620 06:09:23.919445 25630 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.102.20:37051
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935603120653-11672-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935603120653-11672-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935603120653-11672-0/krb5kdc/kudu_127.11.102.2.keytab
--principal=kudu/127.11.102.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.102.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935603120653-11672-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.11.102.2
--webserver_port=0
--tserver_master_addrs=127.11.102.62:41475
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25630
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.102.2
--log_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935603120653-11672-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 05:28:48 UTC on ccycloud.quasar-fixhms.root.comops.site
build id 4457020
I20260620 06:09:23.920456 25630 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:09:23.921422 25630 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 06:09:23.931041 25635 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:09:23.931082 25636 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:09:23.931072 25638 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:09:23.931130 25630 server_base.cc:1034] running on GCE node
I20260620 06:09:23.931761 25630 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:09:23.932374 25630 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:09:23.933569 25630 hybrid_clock.cc:648] HybridClock initialized: now 1781935763933556 us; error 34 us; skew 500 ppm
Jun 20 06:09:23 dist-test-slave-9hm5 krb5kdc[25389](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935763, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.102.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 06:09:23.936244 25630 init.cc:377] Logged in from keytab as kudu/127.11.102.2@KRBTEST.COM (short username kudu)
I20260620 06:09:23.937021 25630 webserver.cc:460] Webserver started at http://127.11.102.2:42277/ using document root <none> and password file <none>
I20260620 06:09:23.937587 25630 fs_manager.cc:359] Metadata directory not provided
I20260620 06:09:23.937686 25630 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:09:23.937896 25630 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 06:09:23.939921 25630 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935603120653-11672-0/minicluster-data/ts-1/data/instance:
uuid: "0d1347dff7a94072ad8cbe2deef6f05a"
format_stamp: "Formatted at 2026-06-20 06:09:23 on dist-test-slave-9hm5"
server_key: "6dd3b8101a03a814b1a68fe753780af8"
server_key_iv: "9bcd1732e4c77ea13bb2b9fadd277fca"
server_key_version: "encryptionkey@0"
I20260620 06:09:23.940462 25630 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935603120653-11672-0/minicluster-data/ts-1/wal/instance:
uuid: "0d1347dff7a94072ad8cbe2deef6f05a"
format_stamp: "Formatted at 2026-06-20 06:09:23 on dist-test-slave-9hm5"
server_key: "6dd3b8101a03a814b1a68fe753780af8"
server_key_iv: "9bcd1732e4c77ea13bb2b9fadd277fca"
server_key_version: "encryptionkey@0"
I20260620 06:09:23.943974 25630 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260620 06:09:23.946254 25645 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:09:23.947376 25630 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 06:09:23.947518 25630 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935603120653-11672-0/minicluster-data/ts-1/data,/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935603120653-11672-0/minicluster-data/ts-1/wal
uuid: "0d1347dff7a94072ad8cbe2deef6f05a"
format_stamp: "Formatted at 2026-06-20 06:09:23 on dist-test-slave-9hm5"
server_key: "6dd3b8101a03a814b1a68fe753780af8"
server_key_iv: "9bcd1732e4c77ea13bb2b9fadd277fca"
server_key_version: "encryptionkey@0"
I20260620 06:09:23.947665 25630 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935603120653-11672-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935603120653-11672-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935603120653-11672-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:09:23.959101 25630 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:09:23.961812 25630 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:09:23.962008 25630 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:09:23.962550 25630 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 06:09:23.963346 25630 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 06:09:23.963426 25630 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 06:09:23.963495 25630 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 06:09:23.963531 25630 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 06:09:23.972074 25630 rpc_server.cc:307] RPC server started. Bound to: 127.11.102.2:36731
I20260620 06:09:23.972138 25758 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.102.2:36731 every 8 connection(s)
I20260620 06:09:23.972960 25630 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935603120653-11672-0/minicluster-data/ts-1/data/info.pb
I20260620 06:09:23.977517 11672 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9HMp43/build/debug/bin/kudu as pid 25630
I20260620 06:09:23.977654 11672 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935603120653-11672-0/minicluster-data/ts-1/wal/instance
I20260620 06:09:23.977949 11672 external_mini_cluster.cc:1428] Setting key 47f9923a3029823e9b8ca5cd795220d2
Jun 20 06:09:23 dist-test-slave-9hm5 krb5kdc[25389](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935763, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.102.2@KRBTEST.COM for kudu/127.11.102.62@KRBTEST.COM
I20260620 06:09:23.988214 25473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:09:23.974664 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.102.2:46591 (local address 127.11.102.62:41475)
0620 06:09:23.974808 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0620 06:09:23.974812 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 06:09:23.975582 (+ 770us) server_negotiation.cc:407] Connection header received
0620 06:09:23.976962 (+ 1380us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:09:23.976965 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:09:23.977023 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:09:23.977103 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:09:23.978620 (+ 1517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:09:23.979398 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:09:23.981181 (+ 1783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:09:23.981597 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:09:23.981744 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:09:23.984121 (+ 2377us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:09:23.984126 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:09:23.984131 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:09:23.984171 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:09:23.985934 (+ 1763us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:09:23.986462 (+ 528us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:09:23.986467 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:09:23.986471 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:09:23.986575 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:09:23.986925 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:09:23.986929 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:09:23.986931 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:09:23.987203 (+ 272us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:09:23.987324 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0620 06:09:23.987845 (+ 521us) server_negotiation.cc:299] Negotiation successful
0620 06:09:23.987996 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
WARNING: no policy specified for kudu/127.11.102.3@KRBTEST.COM; defaulting to no policy
I20260620 06:09:23.988794 25761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:09:23.974996 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.102.62:41475 (local address 127.11.102.2:46591)
0620 06:09:23.975431 (+ 435us) negotiation.cc:107] Waiting for socket to connect
0620 06:09:23.975470 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0620 06:09:23.976748 (+ 1278us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:09:23.977318 (+ 570us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:09:23.977328 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:09:23.977822 (+ 494us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:09:23.978443 (+ 621us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:09:23.978459 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:09:23.979532 (+ 1073us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:09:23.979536 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:09:23.981022 (+ 1486us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:09:23.981036 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:09:23.981760 (+ 724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:09:23.981763 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:09:23.981899 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:09:23.982496 (+ 597us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:09:23.982512 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:09:23.983943 (+ 1431us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:09:23.986060 (+ 2117us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:09:23.986064 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:09:23.986074 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:09:23.986328 (+ 254us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:09:23.986676 (+ 348us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:09:23.986678 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:09:23.986679 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:09:23.986805 (+ 126us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:09:23.987319 (+ 514us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:09:23.987324 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:09:23.987526 (+ 202us) client_negotiation.cc:769] Sending connection context
0620 06:09:23.987731 (+ 205us) client_negotiation.cc:240] Negotiation successful
0620 06:09:23.987963 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":261,"thread_start_us":115,"threads_started":1}
I20260620 06:09:23.989959 25759 heartbeater.cc:347] Connected to a master server at 127.11.102.62:41475
I20260620 06:09:23.990149 25759 heartbeater.cc:464] Registering TS with master...
I20260620 06:09:23.990600 25759 heartbeater.cc:511] Master 127.11.102.62:41475 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.102.3@KRBTEST.COM" created.
I20260620 06:09:23.991746 25434 ts_manager.cc:194] Registered new tserver with Master: 0d1347dff7a94072ad8cbe2deef6f05a (127.11.102.2:36731)
I20260620 06:09:23.992559 25434 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.102.2@KRBTEST.COM'} at 127.11.102.2:46591
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.102.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935603120653-11672-0/krb5kdc/kudu_127.11.102.3.keytab.
Entry for principal kudu/127.11.102.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935603120653-11672-0/krb5kdc/kudu_127.11.102.3.keytab.
I20260620 06:09:24.006358 11672 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9HMp43/build/debug/bin/kudu
/tmp/dist-test-task9HMp43/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935603120653-11672-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935603120653-11672-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935603120653-11672-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935603120653-11672-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.102.3:0
--local_ip_for_outbound_sockets=127.11.102.3
--webserver_interface=127.11.102.3
--webserver_port=0
--tserver_master_addrs=127.11.102.62:41475
--keytab_file=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935603120653-11672-0/krb5kdc/kudu_127.11.102.3.keytab
--principal=kudu/127.11.102.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.102.20:37051
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935603120653-11672-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935603120653-11672-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935603120653-11672-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 06:09:24.114439 25764 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:09:24.114727 25764 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:09:24.114807 25764 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:09:24.119431 25764 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:09:24.119519 25764 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:09:24.119614 25764 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.102.3
I20260620 06:09:24.124743 25764 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.102.20:37051
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935603120653-11672-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935603120653-11672-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935603120653-11672-0/krb5kdc/kudu_127.11.102.3.keytab
--principal=kudu/127.11.102.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.102.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935603120653-11672-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.11.102.3
--webserver_port=0
--tserver_master_addrs=127.11.102.62:41475
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25764
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.102.3
--log_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935603120653-11672-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 05:28:48 UTC on ccycloud.quasar-fixhms.root.comops.site
build id 4457020
I20260620 06:09:24.125800 25764 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:09:24.126840 25764 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 06:09:24.136494 25772 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:09:24.136523 25769 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:09:24.136498 25770 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:09:24.136716 25764 server_base.cc:1034] running on GCE node
I20260620 06:09:24.137295 25764 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:09:24.137939 25764 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:09:24.139133 25764 hybrid_clock.cc:648] HybridClock initialized: now 1781935764139106 us; error 45 us; skew 500 ppm
Jun 20 06:09:24 dist-test-slave-9hm5 krb5kdc[25389](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935764, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.102.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 06:09:24.141925 25764 init.cc:377] Logged in from keytab as kudu/127.11.102.3@KRBTEST.COM (short username kudu)
I20260620 06:09:24.142771 25764 webserver.cc:460] Webserver started at http://127.11.102.3:33315/ using document root <none> and password file <none>
I20260620 06:09:24.143339 25764 fs_manager.cc:359] Metadata directory not provided
I20260620 06:09:24.143440 25764 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:09:24.143738 25764 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 06:09:24.145744 25764 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935603120653-11672-0/minicluster-data/ts-2/data/instance:
uuid: "7c99aa7fa7834c89968adb1265fa3603"
format_stamp: "Formatted at 2026-06-20 06:09:24 on dist-test-slave-9hm5"
server_key: "b3b0ed89c1cc4c95c4ba805a87b5e8c5"
server_key_iv: "57f8a6a192208426f9de3dc1b98d0013"
server_key_version: "encryptionkey@0"
I20260620 06:09:24.146306 25764 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935603120653-11672-0/minicluster-data/ts-2/wal/instance:
uuid: "7c99aa7fa7834c89968adb1265fa3603"
format_stamp: "Formatted at 2026-06-20 06:09:24 on dist-test-slave-9hm5"
server_key: "b3b0ed89c1cc4c95c4ba805a87b5e8c5"
server_key_iv: "57f8a6a192208426f9de3dc1b98d0013"
server_key_version: "encryptionkey@0"
I20260620 06:09:24.150064 25764 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260620 06:09:24.152675 25779 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:09:24.154022 25764 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 06:09:24.154161 25764 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935603120653-11672-0/minicluster-data/ts-2/data,/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935603120653-11672-0/minicluster-data/ts-2/wal
uuid: "7c99aa7fa7834c89968adb1265fa3603"
format_stamp: "Formatted at 2026-06-20 06:09:24 on dist-test-slave-9hm5"
server_key: "b3b0ed89c1cc4c95c4ba805a87b5e8c5"
server_key_iv: "57f8a6a192208426f9de3dc1b98d0013"
server_key_version: "encryptionkey@0"
I20260620 06:09:24.154286 25764 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935603120653-11672-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935603120653-11672-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935603120653-11672-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:09:24.174548 25764 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:09:24.177603 25764 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:09:24.177770 25764 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:09:24.178272 25764 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 06:09:24.179018 25764 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 06:09:24.179064 25764 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 06:09:24.179101 25764 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 06:09:24.179116 25764 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 06:09:24.188180 25764 rpc_server.cc:307] RPC server started. Bound to: 127.11.102.3:43859
I20260620 06:09:24.188225 25892 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.102.3:43859 every 8 connection(s)
I20260620 06:09:24.189072 25764 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935603120653-11672-0/minicluster-data/ts-2/data/info.pb
I20260620 06:09:24.192246 11672 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9HMp43/build/debug/bin/kudu as pid 25764
I20260620 06:09:24.192359 11672 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935603120653-11672-0/minicluster-data/ts-2/wal/instance
I20260620 06:09:24.192662 11672 external_mini_cluster.cc:1428] Setting key 999ac7a3ebe666bfee90aa70ad9fc2ef
Jun 20 06:09:24 dist-test-slave-9hm5 krb5kdc[25389](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935764, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.102.3@KRBTEST.COM for kudu/127.11.102.62@KRBTEST.COM
I20260620 06:09:24.202544 25473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:09:24.190875 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.102.3:57675 (local address 127.11.102.62:41475)
0620 06:09:24.191039 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0620 06:09:24.191045 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 06:09:24.191953 (+ 908us) server_negotiation.cc:407] Connection header received
0620 06:09:24.193448 (+ 1495us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:09:24.193451 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:09:24.193506 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:09:24.193593 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:09:24.194979 (+ 1386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:09:24.195501 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:09:24.196664 (+ 1163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:09:24.196875 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:09:24.196955 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:09:24.199309 (+ 2354us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:09:24.199313 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:09:24.199315 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:09:24.199339 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:09:24.200874 (+ 1535us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:09:24.201304 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:09:24.201307 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:09:24.201308 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:09:24.201381 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:09:24.201662 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:09:24.201664 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:09:24.201665 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:09:24.201829 (+ 164us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:09:24.201901 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0620 06:09:24.202335 (+ 434us) server_negotiation.cc:299] Negotiation successful
0620 06:09:24.202428 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260620 06:09:24.203423 25895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:09:24.191312 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.102.62:41475 (local address 127.11.102.3:57675)
0620 06:09:24.191799 (+ 487us) negotiation.cc:107] Waiting for socket to connect
0620 06:09:24.191840 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0620 06:09:24.193245 (+ 1405us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:09:24.193803 (+ 558us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:09:24.193810 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:09:24.194286 (+ 476us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:09:24.194849 (+ 563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:09:24.194865 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:09:24.195657 (+ 792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:09:24.195663 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:09:24.196553 (+ 890us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:09:24.196562 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:09:24.197002 (+ 440us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:09:24.197004 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:09:24.197140 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:09:24.197673 (+ 533us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:09:24.197695 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:09:24.199153 (+ 1458us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:09:24.200969 (+ 1816us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:09:24.200973 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:09:24.200984 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:09:24.201218 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:09:24.201459 (+ 241us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:09:24.201461 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:09:24.201463 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:09:24.201585 (+ 122us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:09:24.201912 (+ 327us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:09:24.201917 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:09:24.202138 (+ 221us) client_negotiation.cc:769] Sending connection context
0620 06:09:24.202287 (+ 149us) client_negotiation.cc:240] Negotiation successful
0620 06:09:24.202517 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":303,"thread_start_us":133,"threads_started":1}
I20260620 06:09:24.204591 25893 heartbeater.cc:347] Connected to a master server at 127.11.102.62:41475
I20260620 06:09:24.204818 25893 heartbeater.cc:464] Registering TS with master...
I20260620 06:09:24.205371 25893 heartbeater.cc:511] Master 127.11.102.62:41475 requested a full tablet report, sending...
I20260620 06:09:24.206525 25433 ts_manager.cc:194] Registered new tserver with Master: 7c99aa7fa7834c89968adb1265fa3603 (127.11.102.3:43859)
I20260620 06:09:24.207108 25433 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.102.3@KRBTEST.COM'} at 127.11.102.3:57675
I20260620 06:09:24.218776 11672 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 06:09:24.227811 25473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:09:24.220736 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48656 (local address 127.11.102.62:41475)
0620 06:09:24.220868 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0620 06:09:24.220872 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 06:09:24.221030 (+ 158us) server_negotiation.cc:407] Connection header received
0620 06:09:24.221124 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:09:24.221126 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:09:24.221165 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:09:24.221240 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:09:24.221952 (+ 712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:09:24.222443 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:09:24.223232 (+ 789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:09:24.223444 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:09:24.223524 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:09:24.224870 (+ 1346us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:09:24.224873 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:09:24.224876 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:09:24.224898 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:09:24.226419 (+ 1521us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:09:24.226794 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:09:24.226796 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:09:24.226797 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:09:24.226869 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:09:24.227099 (+ 230us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:09:24.227101 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:09:24.227102 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:09:24.227302 (+ 200us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:09:24.227402 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0620 06:09:24.227564 (+ 162us) server_negotiation.cc:299] Negotiation successful
0620 06:09:24.227698 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260620 06:09:24.230639 25433 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48656:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 06:09:24.233933 25433 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 06:09:24.248831 25909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:09:24.243375 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59986 (local address 127.11.102.3:43859)
0620 06:09:24.243833 (+ 458us) server_negotiation.cc:206] Beginning negotiation
0620 06:09:24.243851 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0620 06:09:24.244229 (+ 378us) server_negotiation.cc:407] Connection header received
0620 06:09:24.244385 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:09:24.244395 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:09:24.244529 (+ 134us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:09:24.244700 (+ 171us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 06:09:24.244885 (+ 185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:09:24.245633 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:09:24.247525 (+ 1892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:09:24.248138 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:09:24.248267 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:09:24.248409 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0620 06:09:24.248572 (+ 163us) server_negotiation.cc:299] Negotiation successful
0620 06:09:24.248683 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":356,"thread_start_us":129,"threads_started":1}
I20260620 06:09:24.248839 25904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:09:24.242344 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.102.2:36731 (local address 127.0.0.1:40004)
0620 06:09:24.242828 (+ 484us) negotiation.cc:107] Waiting for socket to connect
0620 06:09:24.242873 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0620 06:09:24.242991 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:09:24.243767 (+ 776us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:09:24.243774 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:09:24.243798 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 06:09:24.243950 (+ 152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:09:24.243957 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:09:24.245653 (+ 1696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:09:24.245659 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:09:24.247132 (+ 1473us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:09:24.247141 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:09:24.248323 (+ 1182us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:09:24.248327 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:09:24.248429 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:09:24.248450 (+ 21us) client_negotiation.cc:769] Sending connection context
0620 06:09:24.248498 (+ 48us) client_negotiation.cc:240] Negotiation successful
0620 06:09:24.248641 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":328,"mutex_wait_us":153,"thread_start_us":76,"threads_started":1}
I20260620 06:09:24.249174 25908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:09:24.243100 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50748 (local address 127.11.102.1:41679)
0620 06:09:24.243538 (+ 438us) server_negotiation.cc:206] Beginning negotiation
0620 06:09:24.243553 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 06:09:24.243574 (+ 21us) server_negotiation.cc:407] Connection header received
0620 06:09:24.243777 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:09:24.243782 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:09:24.243962 (+ 180us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:09:24.244365 (+ 403us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 06:09:24.244500 (+ 135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:09:24.245586 (+ 1086us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:09:24.247517 (+ 1931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:09:24.248317 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:09:24.248411 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:09:24.248550 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0620 06:09:24.248933 (+ 383us) server_negotiation.cc:299] Negotiation successful
0620 06:09:24.249026 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":341,"thread_start_us":82,"threads_started":1}
I20260620 06:09:24.249276 25905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:09:24.242515 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.102.1:41679 (local address 127.0.0.1:50748)
0620 06:09:24.243221 (+ 706us) negotiation.cc:107] Waiting for socket to connect
0620 06:09:24.243234 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 06:09:24.243310 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:09:24.244106 (+ 796us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:09:24.244109 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:09:24.244126 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 06:09:24.244226 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:09:24.244232 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:09:24.245724 (+ 1492us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:09:24.245730 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:09:24.247383 (+ 1653us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:09:24.247394 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:09:24.248611 (+ 1217us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:09:24.248615 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:09:24.248702 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:09:24.248725 (+ 23us) client_negotiation.cc:769] Sending connection context
0620 06:09:24.248794 (+ 69us) client_negotiation.cc:240] Negotiation successful
0620 06:09:24.248858 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":647,"mutex_wait_us":177,"thread_start_us":115,"threads_started":1}
I20260620 06:09:24.248821 25907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:09:24.242977 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.102.3:43859 (local address 127.0.0.1:59986)
0620 06:09:24.244083 (+ 1106us) negotiation.cc:107] Waiting for socket to connect
0620 06:09:24.244098 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 06:09:24.244255 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:09:24.244667 (+ 412us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:09:24.244670 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:09:24.244688 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 06:09:24.244789 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:09:24.244796 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:09:24.245766 (+ 970us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:09:24.245769 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:09:24.247383 (+ 1614us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:09:24.247394 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:09:24.248265 (+ 871us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:09:24.248268 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:09:24.248382 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:09:24.248417 (+ 35us) client_negotiation.cc:769] Sending connection context
0620 06:09:24.248524 (+ 107us) client_negotiation.cc:240] Negotiation successful
0620 06:09:24.248645 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1040,"mutex_wait_us":301,"thread_start_us":81,"threads_started":1}
I20260620 06:09:24.249509 25906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:09:24.242515 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40004 (local address 127.11.102.2:36731)
0620 06:09:24.243209 (+ 694us) server_negotiation.cc:206] Beginning negotiation
0620 06:09:24.243221 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 06:09:24.243249 (+ 28us) server_negotiation.cc:407] Connection header received
0620 06:09:24.243329 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:09:24.243338 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:09:24.243554 (+ 216us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:09:24.244113 (+ 559us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 06:09:24.244283 (+ 170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:09:24.245496 (+ 1213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:09:24.247288 (+ 1792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:09:24.248172 (+ 884us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:09:24.248910 (+ 738us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:09:24.249046 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0620 06:09:24.249108 (+ 62us) server_negotiation.cc:299] Negotiation successful
0620 06:09:24.249197 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":575,"thread_start_us":131,"threads_started":1}
I20260620 06:09:24.252453 25693 tablet_service.cc:1467] Processing CreateTablet for tablet e8daa5c69e82442b9e436139959f91b9 (DEFAULT_TABLE table=test-table [id=d69129a39f9f4ee089af9b6114498a2f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 06:09:24.252525 25827 tablet_service.cc:1467] Processing CreateTablet for tablet e8daa5c69e82442b9e436139959f91b9 (DEFAULT_TABLE table=test-table [id=d69129a39f9f4ee089af9b6114498a2f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 06:09:24.252658 25559 tablet_service.cc:1467] Processing CreateTablet for tablet e8daa5c69e82442b9e436139959f91b9 (DEFAULT_TABLE table=test-table [id=d69129a39f9f4ee089af9b6114498a2f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 06:09:24.253512 25693 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e8daa5c69e82442b9e436139959f91b9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 06:09:24.253518 25559 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e8daa5c69e82442b9e436139959f91b9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 06:09:24.253561 25827 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e8daa5c69e82442b9e436139959f91b9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 06:09:24.259675 25910 tablet_bootstrap.cc:492] T e8daa5c69e82442b9e436139959f91b9 P 0d1347dff7a94072ad8cbe2deef6f05a: Bootstrap starting.
I20260620 06:09:24.261188 25911 tablet_bootstrap.cc:492] T e8daa5c69e82442b9e436139959f91b9 P d7f871fdb6814adabd0538690e2a65a3: Bootstrap starting.
I20260620 06:09:24.262419 25910 tablet_bootstrap.cc:654] T e8daa5c69e82442b9e436139959f91b9 P 0d1347dff7a94072ad8cbe2deef6f05a: Neither blocks nor log segments found. Creating new log.
I20260620 06:09:24.262991 25911 tablet_bootstrap.cc:654] T e8daa5c69e82442b9e436139959f91b9 P d7f871fdb6814adabd0538690e2a65a3: Neither blocks nor log segments found. Creating new log.
I20260620 06:09:24.263316 25910 log.cc:824] T e8daa5c69e82442b9e436139959f91b9 P 0d1347dff7a94072ad8cbe2deef6f05a: Log is configured to *not* fsync() on all Append() calls
I20260620 06:09:24.263329 25912 tablet_bootstrap.cc:492] T e8daa5c69e82442b9e436139959f91b9 P 7c99aa7fa7834c89968adb1265fa3603: Bootstrap starting.
I20260620 06:09:24.264914 25911 log.cc:824] T e8daa5c69e82442b9e436139959f91b9 P d7f871fdb6814adabd0538690e2a65a3: Log is configured to *not* fsync() on all Append() calls
I20260620 06:09:24.265131 25910 tablet_bootstrap.cc:492] T e8daa5c69e82442b9e436139959f91b9 P 0d1347dff7a94072ad8cbe2deef6f05a: No bootstrap required, opened a new log
I20260620 06:09:24.265360 25910 ts_tablet_manager.cc:1397] T e8daa5c69e82442b9e436139959f91b9 P 0d1347dff7a94072ad8cbe2deef6f05a: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.002s
I20260620 06:09:24.265928 25912 tablet_bootstrap.cc:654] T e8daa5c69e82442b9e436139959f91b9 P 7c99aa7fa7834c89968adb1265fa3603: Neither blocks nor log segments found. Creating new log.
I20260620 06:09:24.266732 25912 log.cc:824] T e8daa5c69e82442b9e436139959f91b9 P 7c99aa7fa7834c89968adb1265fa3603: Log is configured to *not* fsync() on all Append() calls
I20260620 06:09:24.266953 25911 tablet_bootstrap.cc:492] T e8daa5c69e82442b9e436139959f91b9 P d7f871fdb6814adabd0538690e2a65a3: No bootstrap required, opened a new log
I20260620 06:09:24.267149 25911 ts_tablet_manager.cc:1397] T e8daa5c69e82442b9e436139959f91b9 P d7f871fdb6814adabd0538690e2a65a3: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260620 06:09:24.268774 25912 tablet_bootstrap.cc:492] T e8daa5c69e82442b9e436139959f91b9 P 7c99aa7fa7834c89968adb1265fa3603: No bootstrap required, opened a new log
I20260620 06:09:24.268956 25912 ts_tablet_manager.cc:1397] T e8daa5c69e82442b9e436139959f91b9 P 7c99aa7fa7834c89968adb1265fa3603: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260620 06:09:24.269068 25910 raft_consensus.cc:348] T e8daa5c69e82442b9e436139959f91b9 P 0d1347dff7a94072ad8cbe2deef6f05a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d1347dff7a94072ad8cbe2deef6f05a" member_type: VOTER last_known_addr { host: "127.11.102.2" port: 36731 } } peers { permanent_uuid: "d7f871fdb6814adabd0538690e2a65a3" member_type: VOTER last_known_addr { host: "127.11.102.1" port: 41679 } } peers { permanent_uuid: "7c99aa7fa7834c89968adb1265fa3603" member_type: VOTER last_known_addr { host: "127.11.102.3" port: 43859 } }
I20260620 06:09:24.269342 25910 raft_consensus.cc:374] T e8daa5c69e82442b9e436139959f91b9 P 0d1347dff7a94072ad8cbe2deef6f05a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 06:09:24.269393 25910 raft_consensus.cc:729] T e8daa5c69e82442b9e436139959f91b9 P 0d1347dff7a94072ad8cbe2deef6f05a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d1347dff7a94072ad8cbe2deef6f05a, State: Initialized, Role: FOLLOWER
I20260620 06:09:24.269861 25910 consensus_queue.cc:260] T e8daa5c69e82442b9e436139959f91b9 P 0d1347dff7a94072ad8cbe2deef6f05a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d1347dff7a94072ad8cbe2deef6f05a" member_type: VOTER last_known_addr { host: "127.11.102.2" port: 36731 } } peers { permanent_uuid: "d7f871fdb6814adabd0538690e2a65a3" member_type: VOTER last_known_addr { host: "127.11.102.1" port: 41679 } } peers { permanent_uuid: "7c99aa7fa7834c89968adb1265fa3603" member_type: VOTER last_known_addr { host: "127.11.102.3" port: 43859 } }
I20260620 06:09:24.270555 25759 heartbeater.cc:503] Master 127.11.102.62:41475 was elected leader, sending a full tablet report...
I20260620 06:09:24.270895 25910 ts_tablet_manager.cc:1428] T e8daa5c69e82442b9e436139959f91b9 P 0d1347dff7a94072ad8cbe2deef6f05a: Time spent starting tablet: real 0.005s user 0.005s sys 0.001s
I20260620 06:09:24.270964 25911 raft_consensus.cc:348] T e8daa5c69e82442b9e436139959f91b9 P d7f871fdb6814adabd0538690e2a65a3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d1347dff7a94072ad8cbe2deef6f05a" member_type: VOTER last_known_addr { host: "127.11.102.2" port: 36731 } } peers { permanent_uuid: "d7f871fdb6814adabd0538690e2a65a3" member_type: VOTER last_known_addr { host: "127.11.102.1" port: 41679 } } peers { permanent_uuid: "7c99aa7fa7834c89968adb1265fa3603" member_type: VOTER last_known_addr { host: "127.11.102.3" port: 43859 } }
I20260620 06:09:24.271246 25911 raft_consensus.cc:374] T e8daa5c69e82442b9e436139959f91b9 P d7f871fdb6814adabd0538690e2a65a3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 06:09:24.271323 25911 raft_consensus.cc:729] T e8daa5c69e82442b9e436139959f91b9 P d7f871fdb6814adabd0538690e2a65a3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d7f871fdb6814adabd0538690e2a65a3, State: Initialized, Role: FOLLOWER
I20260620 06:09:24.271766 25911 consensus_queue.cc:260] T e8daa5c69e82442b9e436139959f91b9 P d7f871fdb6814adabd0538690e2a65a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d1347dff7a94072ad8cbe2deef6f05a" member_type: VOTER last_known_addr { host: "127.11.102.2" port: 36731 } } peers { permanent_uuid: "d7f871fdb6814adabd0538690e2a65a3" member_type: VOTER last_known_addr { host: "127.11.102.1" port: 41679 } } peers { permanent_uuid: "7c99aa7fa7834c89968adb1265fa3603" member_type: VOTER last_known_addr { host: "127.11.102.3" port: 43859 } }
I20260620 06:09:24.272580 25911 ts_tablet_manager.cc:1428] T e8daa5c69e82442b9e436139959f91b9 P d7f871fdb6814adabd0538690e2a65a3: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260620 06:09:24.272933 25625 heartbeater.cc:503] Master 127.11.102.62:41475 was elected leader, sending a full tablet report...
I20260620 06:09:24.272745 25912 raft_consensus.cc:348] T e8daa5c69e82442b9e436139959f91b9 P 7c99aa7fa7834c89968adb1265fa3603 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d1347dff7a94072ad8cbe2deef6f05a" member_type: VOTER last_known_addr { host: "127.11.102.2" port: 36731 } } peers { permanent_uuid: "d7f871fdb6814adabd0538690e2a65a3" member_type: VOTER last_known_addr { host: "127.11.102.1" port: 41679 } } peers { permanent_uuid: "7c99aa7fa7834c89968adb1265fa3603" member_type: VOTER last_known_addr { host: "127.11.102.3" port: 43859 } }
I20260620 06:09:24.273010 25912 raft_consensus.cc:374] T e8daa5c69e82442b9e436139959f91b9 P 7c99aa7fa7834c89968adb1265fa3603 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 06:09:24.273064 25912 raft_consensus.cc:729] T e8daa5c69e82442b9e436139959f91b9 P 7c99aa7fa7834c89968adb1265fa3603 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7c99aa7fa7834c89968adb1265fa3603, State: Initialized, Role: FOLLOWER
I20260620 06:09:24.273483 25912 consensus_queue.cc:260] T e8daa5c69e82442b9e436139959f91b9 P 7c99aa7fa7834c89968adb1265fa3603 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d1347dff7a94072ad8cbe2deef6f05a" member_type: VOTER last_known_addr { host: "127.11.102.2" port: 36731 } } peers { permanent_uuid: "d7f871fdb6814adabd0538690e2a65a3" member_type: VOTER last_known_addr { host: "127.11.102.1" port: 41679 } } peers { permanent_uuid: "7c99aa7fa7834c89968adb1265fa3603" member_type: VOTER last_known_addr { host: "127.11.102.3" port: 43859 } }
I20260620 06:09:24.274087 25893 heartbeater.cc:503] Master 127.11.102.62:41475 was elected leader, sending a full tablet report...
I20260620 06:09:24.274360 25912 ts_tablet_manager.cc:1428] T e8daa5c69e82442b9e436139959f91b9 P 7c99aa7fa7834c89968adb1265fa3603: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260620 06:09:24.333220 25916 raft_consensus.cc:482] T e8daa5c69e82442b9e436139959f91b9 P 0d1347dff7a94072ad8cbe2deef6f05a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 06:09:24.333408 25916 raft_consensus.cc:504] T e8daa5c69e82442b9e436139959f91b9 P 0d1347dff7a94072ad8cbe2deef6f05a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d1347dff7a94072ad8cbe2deef6f05a" member_type: VOTER last_known_addr { host: "127.11.102.2" port: 36731 } } peers { permanent_uuid: "d7f871fdb6814adabd0538690e2a65a3" member_type: VOTER last_known_addr { host: "127.11.102.1" port: 41679 } } peers { permanent_uuid: "7c99aa7fa7834c89968adb1265fa3603" member_type: VOTER last_known_addr { host: "127.11.102.3" port: 43859 } }
I20260620 06:09:24.334601 25916 leader_election.cc:288] T e8daa5c69e82442b9e436139959f91b9 P 0d1347dff7a94072ad8cbe2deef6f05a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d7f871fdb6814adabd0538690e2a65a3 (127.11.102.1:41679), 7c99aa7fa7834c89968adb1265fa3603 (127.11.102.3:43859)
I20260620 06:09:24.338428 25909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:09:24.335016 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.102.2:42953 (local address 127.11.102.3:43859)
0620 06:09:24.335214 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0620 06:09:24.335220 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 06:09:24.335263 (+ 43us) server_negotiation.cc:407] Connection header received
0620 06:09:24.335424 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:09:24.335427 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:09:24.335478 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:09:24.335564 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 06:09:24.335850 (+ 286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:09:24.336351 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:09:24.337382 (+ 1031us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:09:24.337833 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:09:24.337967 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:09:24.338017 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0620 06:09:24.338212 (+ 195us) server_negotiation.cc:299] Negotiation successful
0620 06:09:24.338274 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":102}
I20260620 06:09:24.338441 25908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:09:24.334977 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.102.2:42923 (local address 127.11.102.1:41679)
0620 06:09:24.335196 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0620 06:09:24.335200 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 06:09:24.335213 (+ 13us) server_negotiation.cc:407] Connection header received
0620 06:09:24.335260 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:09:24.335262 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:09:24.335306 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:09:24.335419 (+ 113us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 06:09:24.335750 (+ 331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:09:24.336259 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:09:24.337354 (+ 1095us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:09:24.337774 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:09:24.337920 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:09:24.337980 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0620 06:09:24.338222 (+ 242us) server_negotiation.cc:299] Negotiation successful
0620 06:09:24.338289 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":100}
I20260620 06:09:24.338446 25761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:09:24.334794 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.102.1:41679 (local address 127.11.102.2:42923)
0620 06:09:24.334936 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0620 06:09:24.334955 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 06:09:24.335055 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:09:24.335424 (+ 369us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:09:24.335427 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:09:24.335446 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 06:09:24.335576 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:09:24.335582 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:09:24.336390 (+ 808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:09:24.336395 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:09:24.337207 (+ 812us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:09:24.337217 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:09:24.337907 (+ 690us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:09:24.337911 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:09:24.338023 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:09:24.338089 (+ 66us) client_negotiation.cc:769] Sending connection context
0620 06:09:24.338204 (+ 115us) client_negotiation.cc:240] Negotiation successful
0620 06:09:24.338281 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":56,"mutex_wait_us":39}
I20260620 06:09:24.338461 25919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:09:24.334856 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.102.3:43859 (local address 127.11.102.2:42953)
0620 06:09:24.335187 (+ 331us) negotiation.cc:107] Waiting for socket to connect
0620 06:09:24.335202 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 06:09:24.335310 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:09:24.335585 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:09:24.335588 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:09:24.335604 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 06:09:24.335744 (+ 140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:09:24.335749 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:09:24.336469 (+ 720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:09:24.336475 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:09:24.337260 (+ 785us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:09:24.337268 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:09:24.337957 (+ 689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:09:24.337961 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:09:24.338088 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:09:24.338105 (+ 17us) client_negotiation.cc:769] Sending connection context
0620 06:09:24.338199 (+ 94us) client_negotiation.cc:240] Negotiation successful
0620 06:09:24.338273 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":264,"mutex_wait_us":29,"thread_start_us":127,"threads_started":1}
I20260620 06:09:24.339251 25579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8daa5c69e82442b9e436139959f91b9" candidate_uuid: "0d1347dff7a94072ad8cbe2deef6f05a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d7f871fdb6814adabd0538690e2a65a3" is_pre_election: true
I20260620 06:09:24.339321 25847 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8daa5c69e82442b9e436139959f91b9" candidate_uuid: "0d1347dff7a94072ad8cbe2deef6f05a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7c99aa7fa7834c89968adb1265fa3603" is_pre_election: true
I20260620 06:09:24.339573 25579 raft_consensus.cc:2454] T e8daa5c69e82442b9e436139959f91b9 P d7f871fdb6814adabd0538690e2a65a3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d1347dff7a94072ad8cbe2deef6f05a in term 0.
I20260620 06:09:24.339730 25847 raft_consensus.cc:2454] T e8daa5c69e82442b9e436139959f91b9 P 7c99aa7fa7834c89968adb1265fa3603 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d1347dff7a94072ad8cbe2deef6f05a in term 0.
I20260620 06:09:24.340171 25646 leader_election.cc:302] T e8daa5c69e82442b9e436139959f91b9 P 0d1347dff7a94072ad8cbe2deef6f05a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 0d1347dff7a94072ad8cbe2deef6f05a, 7c99aa7fa7834c89968adb1265fa3603, d7f871fdb6814adabd0538690e2a65a3; no voters:
I20260620 06:09:24.340451 25916 raft_consensus.cc:2781] T e8daa5c69e82442b9e436139959f91b9 P 0d1347dff7a94072ad8cbe2deef6f05a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 06:09:24.340533 25916 raft_consensus.cc:482] T e8daa5c69e82442b9e436139959f91b9 P 0d1347dff7a94072ad8cbe2deef6f05a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 06:09:24.340591 25916 raft_consensus.cc:3037] T e8daa5c69e82442b9e436139959f91b9 P 0d1347dff7a94072ad8cbe2deef6f05a [term 0 FOLLOWER]: Advancing to term 1
I20260620 06:09:24.341670 25916 raft_consensus.cc:504] T e8daa5c69e82442b9e436139959f91b9 P 0d1347dff7a94072ad8cbe2deef6f05a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d1347dff7a94072ad8cbe2deef6f05a" member_type: VOTER last_known_addr { host: "127.11.102.2" port: 36731 } } peers { permanent_uuid: "d7f871fdb6814adabd0538690e2a65a3" member_type: VOTER last_known_addr { host: "127.11.102.1" port: 41679 } } peers { permanent_uuid: "7c99aa7fa7834c89968adb1265fa3603" member_type: VOTER last_known_addr { host: "127.11.102.3" port: 43859 } }
I20260620 06:09:24.342038 25916 leader_election.cc:288] T e8daa5c69e82442b9e436139959f91b9 P 0d1347dff7a94072ad8cbe2deef6f05a [CANDIDATE]: Term 1 election: Requested vote from peers d7f871fdb6814adabd0538690e2a65a3 (127.11.102.1:41679), 7c99aa7fa7834c89968adb1265fa3603 (127.11.102.3:43859)
I20260620 06:09:24.342423 25579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8daa5c69e82442b9e436139959f91b9" candidate_uuid: "0d1347dff7a94072ad8cbe2deef6f05a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d7f871fdb6814adabd0538690e2a65a3"
I20260620 06:09:24.342557 25847 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8daa5c69e82442b9e436139959f91b9" candidate_uuid: "0d1347dff7a94072ad8cbe2deef6f05a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7c99aa7fa7834c89968adb1265fa3603"
I20260620 06:09:24.342569 25579 raft_consensus.cc:3037] T e8daa5c69e82442b9e436139959f91b9 P d7f871fdb6814adabd0538690e2a65a3 [term 0 FOLLOWER]: Advancing to term 1
I20260620 06:09:24.342710 25847 raft_consensus.cc:3037] T e8daa5c69e82442b9e436139959f91b9 P 7c99aa7fa7834c89968adb1265fa3603 [term 0 FOLLOWER]: Advancing to term 1
I20260620 06:09:24.343595 25579 raft_consensus.cc:2454] T e8daa5c69e82442b9e436139959f91b9 P d7f871fdb6814adabd0538690e2a65a3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d1347dff7a94072ad8cbe2deef6f05a in term 1.
I20260620 06:09:24.343914 25847 raft_consensus.cc:2454] T e8daa5c69e82442b9e436139959f91b9 P 7c99aa7fa7834c89968adb1265fa3603 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d1347dff7a94072ad8cbe2deef6f05a in term 1.
I20260620 06:09:24.343940 25646 leader_election.cc:302] T e8daa5c69e82442b9e436139959f91b9 P 0d1347dff7a94072ad8cbe2deef6f05a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d1347dff7a94072ad8cbe2deef6f05a, d7f871fdb6814adabd0538690e2a65a3; no voters:
I20260620 06:09:24.344202 25916 raft_consensus.cc:2781] T e8daa5c69e82442b9e436139959f91b9 P 0d1347dff7a94072ad8cbe2deef6f05a [term 1 FOLLOWER]: Leader election won for term 1
I20260620 06:09:24.344437 25916 raft_consensus.cc:686] T e8daa5c69e82442b9e436139959f91b9 P 0d1347dff7a94072ad8cbe2deef6f05a [term 1 LEADER]: Becoming Leader. State: Replica: 0d1347dff7a94072ad8cbe2deef6f05a, State: Running, Role: LEADER
I20260620 06:09:24.344770 25916 consensus_queue.cc:237] T e8daa5c69e82442b9e436139959f91b9 P 0d1347dff7a94072ad8cbe2deef6f05a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d1347dff7a94072ad8cbe2deef6f05a" member_type: VOTER last_known_addr { host: "127.11.102.2" port: 36731 } } peers { permanent_uuid: "d7f871fdb6814adabd0538690e2a65a3" member_type: VOTER last_known_addr { host: "127.11.102.1" port: 41679 } } peers { permanent_uuid: "7c99aa7fa7834c89968adb1265fa3603" member_type: VOTER last_known_addr { host: "127.11.102.3" port: 43859 } }
I20260620 06:09:24.347790 25434 catalog_manager.cc:5495] T e8daa5c69e82442b9e436139959f91b9 P 0d1347dff7a94072ad8cbe2deef6f05a reported cstate change: term changed from 0 to 1, leader changed from <none> to 0d1347dff7a94072ad8cbe2deef6f05a (127.11.102.2). New cstate: current_term: 1 leader_uuid: "0d1347dff7a94072ad8cbe2deef6f05a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d1347dff7a94072ad8cbe2deef6f05a" member_type: VOTER last_known_addr { host: "127.11.102.2" port: 36731 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d7f871fdb6814adabd0538690e2a65a3" member_type: VOTER last_known_addr { host: "127.11.102.1" port: 41679 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7c99aa7fa7834c89968adb1265fa3603" member_type: VOTER last_known_addr { host: "127.11.102.3" port: 43859 } health_report { overall_health: UNKNOWN } } }
I20260620 06:09:24.380800 25906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:09:24.377345 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40018 (local address 127.11.102.2:36731)
0620 06:09:24.377476 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0620 06:09:24.377479 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 06:09:24.377497 (+ 18us) server_negotiation.cc:407] Connection header received
0620 06:09:24.377578 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:09:24.377580 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:09:24.377633 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:09:24.377754 (+ 121us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 06:09:24.378021 (+ 267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:09:24.378524 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:09:24.379322 (+ 798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:09:24.379522 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:09:24.379595 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:09:24.379923 (+ 328us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 06:09:24.380321 (+ 398us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 06:09:24.380410 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0620 06:09:24.380617 (+ 207us) server_negotiation.cc:299] Negotiation successful
0620 06:09:24.380669 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260620 06:09:24.386605 25579 raft_consensus.cc:1261] T e8daa5c69e82442b9e436139959f91b9 P d7f871fdb6814adabd0538690e2a65a3 [term 1 FOLLOWER]: Refusing update from remote peer 0d1347dff7a94072ad8cbe2deef6f05a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 06:09:24.386605 25847 raft_consensus.cc:1261] T e8daa5c69e82442b9e436139959f91b9 P 7c99aa7fa7834c89968adb1265fa3603 [term 1 FOLLOWER]: Refusing update from remote peer 0d1347dff7a94072ad8cbe2deef6f05a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 06:09:24.387399 25920 consensus_queue.cc:1035] T e8daa5c69e82442b9e436139959f91b9 P 0d1347dff7a94072ad8cbe2deef6f05a [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7f871fdb6814adabd0538690e2a65a3" member_type: VOTER last_known_addr { host: "127.11.102.1" port: 41679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 06:09:24.387612 25916 consensus_queue.cc:1035] T e8daa5c69e82442b9e436139959f91b9 P 0d1347dff7a94072ad8cbe2deef6f05a [LEADER]: Connected to new peer: Peer: permanent_uuid: "7c99aa7fa7834c89968adb1265fa3603" member_type: VOTER last_known_addr { host: "127.11.102.3" port: 43859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 06:09:24.397125 25925 mvcc.cc:205] Tried to move back new op lower bound from 7298808890920292352 to 7298808890758782976. Current Snapshot: MvccSnapshot[applied={T|T < 7298808890920292352}]
I20260620 06:09:24.402482 25433 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48656:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 06:09:24.402669 25433 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48656:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 06:09:24.405158 25433 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6743bd85c057436695ea713d0e6cd57b: Sending DeleteTablet for 3 replicas of tablet e8daa5c69e82442b9e436139959f91b9
I20260620 06:09:24.405911 25693 tablet_service.cc:1514] Processing DeleteTablet for tablet e8daa5c69e82442b9e436139959f91b9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 06:09:24 UTC) from {username='kudu', principal='kudu/127.11.102.62@KRBTEST.COM'} at 127.0.0.1:40004
I20260620 06:09:24.405980 25559 tablet_service.cc:1514] Processing DeleteTablet for tablet e8daa5c69e82442b9e436139959f91b9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 06:09:24 UTC) from {username='kudu', principal='kudu/127.11.102.62@KRBTEST.COM'} at 127.0.0.1:50748
I20260620 06:09:24.406162 11672 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9HMp43/build/debug/bin/kudu with pid 25496
I20260620 06:09:24.406128 25827 tablet_service.cc:1514] Processing DeleteTablet for tablet e8daa5c69e82442b9e436139959f91b9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 06:09:24 UTC) from {username='kudu', principal='kudu/127.11.102.62@KRBTEST.COM'} at 127.0.0.1:59986
I20260620 06:09:24.407151 25932 tablet_replica.cc:330] T e8daa5c69e82442b9e436139959f91b9 P 0d1347dff7a94072ad8cbe2deef6f05a: stopping tablet replica
I20260620 06:09:24.407475 25933 tablet_replica.cc:330] T e8daa5c69e82442b9e436139959f91b9 P 7c99aa7fa7834c89968adb1265fa3603: stopping tablet replica
I20260620 06:09:24.407478 25932 raft_consensus.cc:2229] T e8daa5c69e82442b9e436139959f91b9 P 0d1347dff7a94072ad8cbe2deef6f05a [term 1 LEADER]: Raft consensus shutting down.
I20260620 06:09:24.407809 25933 raft_consensus.cc:2229] T e8daa5c69e82442b9e436139959f91b9 P 7c99aa7fa7834c89968adb1265fa3603 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 06:09:24.408046 25932 raft_consensus.cc:2258] T e8daa5c69e82442b9e436139959f91b9 P 0d1347dff7a94072ad8cbe2deef6f05a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 06:09:24.408128 25933 raft_consensus.cc:2258] T e8daa5c69e82442b9e436139959f91b9 P 7c99aa7fa7834c89968adb1265fa3603 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 06:09:24.409612 25932 ts_tablet_manager.cc:1905] T e8daa5c69e82442b9e436139959f91b9 P 0d1347dff7a94072ad8cbe2deef6f05a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 06:09:24.409612 25933 ts_tablet_manager.cc:1905] T e8daa5c69e82442b9e436139959f91b9 P 7c99aa7fa7834c89968adb1265fa3603: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 06:09:24.412469 25932 ts_tablet_manager.cc:1918] T e8daa5c69e82442b9e436139959f91b9 P 0d1347dff7a94072ad8cbe2deef6f05a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 06:09:24.412469 25933 ts_tablet_manager.cc:1918] T e8daa5c69e82442b9e436139959f91b9 P 7c99aa7fa7834c89968adb1265fa3603: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 06:09:24.412572 25933 log.cc:1196] T e8daa5c69e82442b9e436139959f91b9 P 7c99aa7fa7834c89968adb1265fa3603: Deleting WAL directory at /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935603120653-11672-0/minicluster-data/ts-2/wal/wals/e8daa5c69e82442b9e436139959f91b9
I20260620 06:09:24.412686 25932 log.cc:1196] T e8daa5c69e82442b9e436139959f91b9 P 0d1347dff7a94072ad8cbe2deef6f05a: Deleting WAL directory at /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935603120653-11672-0/minicluster-data/ts-1/wal/wals/e8daa5c69e82442b9e436139959f91b9
I20260620 06:09:24.413053 25933 ts_tablet_manager.cc:1939] T e8daa5c69e82442b9e436139959f91b9 P 7c99aa7fa7834c89968adb1265fa3603: Deleting consensus metadata
I20260620 06:09:24.413143 25932 ts_tablet_manager.cc:1939] T e8daa5c69e82442b9e436139959f91b9 P 0d1347dff7a94072ad8cbe2deef6f05a: Deleting consensus metadata
W20260620 06:09:24.413789 25418 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.11.102.1:41679 (error 108)
I20260620 06:09:24.414355 25421 catalog_manager.cc:4841] TS 0d1347dff7a94072ad8cbe2deef6f05a (127.11.102.2:36731): tablet e8daa5c69e82442b9e436139959f91b9 (table test-table [id=d69129a39f9f4ee089af9b6114498a2f]) successfully deleted
I20260620 06:09:24.414356 25420 catalog_manager.cc:4841] TS 7c99aa7fa7834c89968adb1265fa3603 (127.11.102.3:43859): tablet e8daa5c69e82442b9e436139959f91b9 (table test-table [id=d69129a39f9f4ee089af9b6114498a2f]) successfully deleted
I20260620 06:09:24.414709 11672 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9HMp43/build/debug/bin/kudu with pid 25630
I20260620 06:09:24.415314 25907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:09:24.414995 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.102.1:41679 (local address 127.0.0.1:50756)
0620 06:09:24.415166 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0620 06:09:24.415231 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.102.1:41679: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96}
W20260620 06:09:24.415477 25418 catalog_manager.cc:4584] TS d7f871fdb6814adabd0538690e2a65a3 (127.11.102.1:41679): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet e8daa5c69e82442b9e436139959f91b9: Network error: Client connection negotiation failed: client connection to 127.11.102.1:41679: connect: Connection refused (error 111)
I20260620 06:09:24.420794 11672 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9HMp43/build/debug/bin/kudu with pid 25764
I20260620 06:09:24.426481 11672 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9HMp43/build/debug/bin/kudu with pid 25402
I20260620 06:09:24.433729 11672 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9HMp43/build/debug/bin/kudu
/tmp/dist-test-task9HMp43/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935603120653-11672-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935603120653-11672-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935603120653-11672-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935603120653-11672-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.102.62:41475
--webserver_interface=127.11.102.62
--webserver_port=43041
--keytab_file=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935603120653-11672-0/krb5kdc/kudu_127.11.102.62.keytab
--principal=kudu/127.11.102.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.102.20:37051
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.102.62:41475
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935603120653-11672-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935603120653-11672-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935603120653-11672-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 06:09:24.543669 25935 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:09:24.543928 25935 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:09:24.543969 25935 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:09:24.547978 25935 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 06:09:24.548036 25935 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:09:24.548056 25935 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:09:24.548074 25935 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 06:09:24.548089 25935 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 06:09:24.553321 25935 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.102.20:37051
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935603120653-11672-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935603120653-11672-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.11.102.62:41475
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935603120653-11672-0/krb5kdc/kudu_127.11.102.62.keytab
--principal=kudu/127.11.102.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.102.62:41475
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935603120653-11672-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.102.62
--webserver_port=43041
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25935
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935603120653-11672-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 05:28:48 UTC on ccycloud.quasar-fixhms.root.comops.site
build id 4457020
I20260620 06:09:24.554313 25935 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:09:24.555328 25935 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 06:09:24.568089 25942 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:09:24.568163 25941 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:09:24.568125 25944 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:09:24.568141 25935 server_base.cc:1034] running on GCE node
I20260620 06:09:24.569020 25935 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:09:24.570214 25935 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:09:24.571410 25935 hybrid_clock.cc:648] HybridClock initialized: now 1781935764571390 us; error 40 us; skew 500 ppm
Jun 20 06:09:24 dist-test-slave-9hm5 krb5kdc[25389](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935764, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.102.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 06:09:24.574361 25935 init.cc:377] Logged in from keytab as kudu/127.11.102.62@KRBTEST.COM (short username kudu)
I20260620 06:09:24.575206 25935 webserver.cc:460] Webserver started at http://127.11.102.62:43041/ using document root <none> and password file <none>
I20260620 06:09:24.575855 25935 fs_manager.cc:359] Metadata directory not provided
I20260620 06:09:24.575929 25935 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:09:24.579756 25935 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 06:09:24.581959 25951 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:09:24.583191 25935 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 06:09:24.583344 25935 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935603120653-11672-0/minicluster-data/master-0/data,/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935603120653-11672-0/minicluster-data/master-0/wal
uuid: "6743bd85c057436695ea713d0e6cd57b"
format_stamp: "Formatted at 2026-06-20 06:09:23 on dist-test-slave-9hm5"
server_key: "bc428878fb4faafd9bd236886d35abf7"
server_key_iv: "aa844420908e3443787e4f2fb6edd13e"
server_key_version: "encryptionkey@0"
I20260620 06:09:24.584038 25935 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935603120653-11672-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935603120653-11672-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935603120653-11672-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:09:24.601531 25935 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:09:24.604893 25935 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:09:24.605105 25935 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:09:24.612123 25935 rpc_server.cc:307] RPC server started. Bound to: 127.11.102.62:41475
I20260620 06:09:24.612404 26003 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.102.62:41475 every 8 connection(s)
I20260620 06:09:24.613242 25935 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935603120653-11672-0/minicluster-data/master-0/data/info.pb
I20260620 06:09:24.617575 26004 sys_catalog.cc:263] Verifying existing consensus state
I20260620 06:09:24.619273 26004 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6743bd85c057436695ea713d0e6cd57b: Bootstrap starting.
I20260620 06:09:24.620299 11672 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9HMp43/build/debug/bin/kudu as pid 25935
I20260620 06:09:24.620718 11672 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9HMp43/build/debug/bin/kudu
/tmp/dist-test-task9HMp43/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935603120653-11672-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935603120653-11672-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935603120653-11672-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935603120653-11672-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.102.1:41679
--local_ip_for_outbound_sockets=127.11.102.1
--tserver_master_addrs=127.11.102.62:41475
--webserver_port=42581
--webserver_interface=127.11.102.1
--keytab_file=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935603120653-11672-0/krb5kdc/kudu_127.11.102.1.keytab
--principal=kudu/127.11.102.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.102.20:37051
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935603120653-11672-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935603120653-11672-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935603120653-11672-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 06:09:24.636411 26004 log.cc:824] T 00000000000000000000000000000000 P 6743bd85c057436695ea713d0e6cd57b: Log is configured to *not* fsync() on all Append() calls
I20260620 06:09:24.645179 26004 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6743bd85c057436695ea713d0e6cd57b: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 06:09:24.645546 26004 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6743bd85c057436695ea713d0e6cd57b: Bootstrap complete.
I20260620 06:09:24.649636 26004 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6743bd85c057436695ea713d0e6cd57b [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6743bd85c057436695ea713d0e6cd57b" member_type: VOTER last_known_addr { host: "127.11.102.62" port: 41475 } }
I20260620 06:09:24.650074 26004 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6743bd85c057436695ea713d0e6cd57b [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6743bd85c057436695ea713d0e6cd57b, State: Initialized, Role: FOLLOWER
I20260620 06:09:24.650555 26004 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6743bd85c057436695ea713d0e6cd57b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6743bd85c057436695ea713d0e6cd57b" member_type: VOTER last_known_addr { host: "127.11.102.62" port: 41475 } }
I20260620 06:09:24.650710 26004 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6743bd85c057436695ea713d0e6cd57b [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 06:09:24.650831 26004 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6743bd85c057436695ea713d0e6cd57b [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 06:09:24.650970 26004 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6743bd85c057436695ea713d0e6cd57b [term 1 FOLLOWER]: Advancing to term 2
I20260620 06:09:24.652984 26004 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6743bd85c057436695ea713d0e6cd57b [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6743bd85c057436695ea713d0e6cd57b" member_type: VOTER last_known_addr { host: "127.11.102.62" port: 41475 } }
I20260620 06:09:24.653368 26004 leader_election.cc:302] T 00000000000000000000000000000000 P 6743bd85c057436695ea713d0e6cd57b [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6743bd85c057436695ea713d0e6cd57b; no voters:
I20260620 06:09:24.653740 26004 leader_election.cc:288] T 00000000000000000000000000000000 P 6743bd85c057436695ea713d0e6cd57b [CANDIDATE]: Term 2 election: Requested vote from peers
I20260620 06:09:24.653821 26008 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6743bd85c057436695ea713d0e6cd57b [term 2 FOLLOWER]: Leader election won for term 2
I20260620 06:09:24.654062 26008 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6743bd85c057436695ea713d0e6cd57b [term 2 LEADER]: Becoming Leader. State: Replica: 6743bd85c057436695ea713d0e6cd57b, State: Running, Role: LEADER
I20260620 06:09:24.654425 26008 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6743bd85c057436695ea713d0e6cd57b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6743bd85c057436695ea713d0e6cd57b" member_type: VOTER last_known_addr { host: "127.11.102.62" port: 41475 } }
I20260620 06:09:24.655030 26004 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6743bd85c057436695ea713d0e6cd57b [sys.catalog]: configured and running, proceeding with master startup.
I20260620 06:09:24.655885 26009 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6743bd85c057436695ea713d0e6cd57b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "6743bd85c057436695ea713d0e6cd57b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6743bd85c057436695ea713d0e6cd57b" member_type: VOTER last_known_addr { host: "127.11.102.62" port: 41475 } } }
I20260620 06:09:24.656028 26009 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6743bd85c057436695ea713d0e6cd57b [sys.catalog]: This master's current role is: LEADER
I20260620 06:09:24.656325 26010 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6743bd85c057436695ea713d0e6cd57b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6743bd85c057436695ea713d0e6cd57b. Latest consensus state: current_term: 2 leader_uuid: "6743bd85c057436695ea713d0e6cd57b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6743bd85c057436695ea713d0e6cd57b" member_type: VOTER last_known_addr { host: "127.11.102.62" port: 41475 } } }
I20260620 06:09:24.656473 26010 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6743bd85c057436695ea713d0e6cd57b [sys.catalog]: This master's current role is: LEADER
I20260620 06:09:24.656958 26014 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 06:09:24.661626 26014 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 06:09:24.662493 26014 catalog_manager.cc:1259] Loaded cluster ID: c8554031cfe44467b49feb636b18c097
I20260620 06:09:24.662552 26014 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 06:09:24.663954 26014 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260620 06:09:24.668578 26014 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260620 06:09:24.673794 26014 catalog_manager.cc:1504] Loading token signing keys...
I20260620 06:09:24.674537 26014 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 6743bd85c057436695ea713d0e6cd57b: Loaded TSK: 0
I20260620 06:09:24.675798 26014 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260620 06:09:24.735971 26006 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:09:24.736227 26006 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:09:24.736269 26006 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:09:24.740267 26006 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:09:24.740321 26006 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:09:24.740379 26006 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.102.1
I20260620 06:09:24.745549 26006 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.102.20:37051
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935603120653-11672-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935603120653-11672-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935603120653-11672-0/krb5kdc/kudu_127.11.102.1.keytab
--principal=kudu/127.11.102.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.102.1:41679
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935603120653-11672-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.11.102.1
--webserver_port=42581
--tserver_master_addrs=127.11.102.62:41475
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26006
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.102.1
--log_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935603120653-11672-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 05:28:48 UTC on ccycloud.quasar-fixhms.root.comops.site
build id 4457020
I20260620 06:09:24.746524 26006 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:09:24.747499 26006 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 06:09:24.757861 26031 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:09:24.757901 26032 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:09:24.758299 26034 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:09:24.758373 26006 server_base.cc:1034] running on GCE node
I20260620 06:09:24.758792 26006 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:09:24.759446 26006 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:09:24.760630 26006 hybrid_clock.cc:648] HybridClock initialized: now 1781935764760601 us; error 50 us; skew 500 ppm
Jun 20 06:09:24 dist-test-slave-9hm5 krb5kdc[25389](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935764, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.102.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 06:09:24.763257 26006 init.cc:377] Logged in from keytab as kudu/127.11.102.1@KRBTEST.COM (short username kudu)
I20260620 06:09:24.764237 26006 webserver.cc:460] Webserver started at http://127.11.102.1:42581/ using document root <none> and password file <none>
I20260620 06:09:24.764775 26006 fs_manager.cc:359] Metadata directory not provided
I20260620 06:09:24.764881 26006 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:09:24.768750 26006 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260620 06:09:24.771060 26041 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:09:24.772382 26006 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 06:09:24.772519 26006 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935603120653-11672-0/minicluster-data/ts-0/data,/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935603120653-11672-0/minicluster-data/ts-0/wal
uuid: "d7f871fdb6814adabd0538690e2a65a3"
format_stamp: "Formatted at 2026-06-20 06:09:23 on dist-test-slave-9hm5"
server_key: "110aac0377228e58ef7c06a5891315b6"
server_key_iv: "389c4dcb9928faf9e8cd6baa4e0b02f5"
server_key_version: "encryptionkey@0"
I20260620 06:09:24.773063 26006 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935603120653-11672-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935603120653-11672-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935603120653-11672-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:09:24.784680 26006 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:09:24.788286 26006 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:09:24.788496 26006 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:09:24.789098 26006 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 06:09:24.790292 26048 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260620 06:09:24.795442 26006 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260620 06:09:24.795532 26006 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.005s user 0.000s sys 0.000s
I20260620 06:09:24.795631 26006 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260620 06:09:24.797405 26006 ts_tablet_manager.cc:610] Registered 1 tablets
I20260620 06:09:24.797482 26006 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260620 06:09:24.797734 26048 tablet_bootstrap.cc:492] T e8daa5c69e82442b9e436139959f91b9 P d7f871fdb6814adabd0538690e2a65a3: Bootstrap starting.
I20260620 06:09:24.807981 26006 rpc_server.cc:307] RPC server started. Bound to: 127.11.102.1:41679
I20260620 06:09:24.808058 26155 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.102.1:41679 every 8 connection(s)
I20260620 06:09:24.808940 26006 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935603120653-11672-0/minicluster-data/ts-0/data/info.pb
I20260620 06:09:24.817301 11672 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9HMp43/build/debug/bin/kudu as pid 26006
I20260620 06:09:24.817798 11672 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9HMp43/build/debug/bin/kudu
/tmp/dist-test-task9HMp43/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935603120653-11672-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935603120653-11672-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935603120653-11672-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935603120653-11672-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.102.2:36731
--local_ip_for_outbound_sockets=127.11.102.2
--tserver_master_addrs=127.11.102.62:41475
--webserver_port=42277
--webserver_interface=127.11.102.2
--keytab_file=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935603120653-11672-0/krb5kdc/kudu_127.11.102.2.keytab
--principal=kudu/127.11.102.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.102.20:37051
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935603120653-11672-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935603120653-11672-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935603120653-11672-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 06:09:24.818094 26048 log.cc:824] T e8daa5c69e82442b9e436139959f91b9 P d7f871fdb6814adabd0538690e2a65a3: Log is configured to *not* fsync() on all Append() calls
Jun 20 06:09:24 dist-test-slave-9hm5 krb5kdc[25389](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935764, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.102.1@KRBTEST.COM for kudu/127.11.102.62@KRBTEST.COM
I20260620 06:09:24.825992 26048 tablet_bootstrap.cc:492] T e8daa5c69e82442b9e436139959f91b9 P d7f871fdb6814adabd0538690e2a65a3: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 06:09:24.826452 26048 tablet_bootstrap.cc:492] T e8daa5c69e82442b9e436139959f91b9 P d7f871fdb6814adabd0538690e2a65a3: Bootstrap complete.
I20260620 06:09:24.827792 26048 ts_tablet_manager.cc:1397] T e8daa5c69e82442b9e436139959f91b9 P d7f871fdb6814adabd0538690e2a65a3: Time spent bootstrapping tablet: real 0.030s user 0.014s sys 0.015s
I20260620 06:09:24.828121 26159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:09:24.811163 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.102.62:41475 (local address 127.11.102.1:33629)
0620 06:09:24.811809 (+ 646us) negotiation.cc:107] Waiting for socket to connect
0620 06:09:24.811861 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0620 06:09:24.812649 (+ 788us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:09:24.814023 (+ 1374us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:09:24.814029 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:09:24.814373 (+ 344us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:09:24.814810 (+ 437us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:09:24.814823 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:09:24.816764 (+ 1941us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:09:24.816767 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:09:24.818072 (+ 1305us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:09:24.818086 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:09:24.818962 (+ 876us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:09:24.818966 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:09:24.819195 (+ 229us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:09:24.819952 (+ 757us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:09:24.819977 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:09:24.821762 (+ 1785us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:09:24.824536 (+ 2774us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:09:24.824541 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:09:24.824553 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:09:24.824891 (+ 338us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:09:24.825325 (+ 434us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:09:24.825328 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:09:24.825330 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:09:24.825489 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:09:24.826253 (+ 764us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:09:24.826261 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:09:24.826547 (+ 286us) client_negotiation.cc:769] Sending connection context
0620 06:09:24.826750 (+ 203us) client_negotiation.cc:240] Negotiation successful
0620 06:09:24.827088 (+ 338us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":441,"thread_start_us":140,"threads_started":1}
I20260620 06:09:24.828281 26158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:09:24.811246 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.102.1:33629 (local address 127.11.102.62:41475)
0620 06:09:24.811736 (+ 490us) server_negotiation.cc:206] Beginning negotiation
0620 06:09:24.811759 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0620 06:09:24.812000 (+ 241us) server_negotiation.cc:407] Connection header received
0620 06:09:24.813147 (+ 1147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:09:24.813171 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:09:24.813581 (+ 410us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:09:24.813963 (+ 382us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:09:24.814996 (+ 1033us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:09:24.816609 (+ 1613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:09:24.818209 (+ 1600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:09:24.818805 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:09:24.818998 (+ 193us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:09:24.821961 (+ 2963us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:09:24.821970 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:09:24.821987 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:09:24.822024 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:09:24.824394 (+ 2370us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:09:24.825027 (+ 633us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:09:24.825036 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:09:24.825043 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:09:24.825206 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:09:24.825603 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:09:24.825608 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:09:24.825612 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:09:24.826121 (+ 509us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:09:24.826271 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0620 06:09:24.826942 (+ 671us) server_negotiation.cc:299] Negotiation successful
0620 06:09:24.827278 (+ 336us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":267,"thread_start_us":128,"threads_started":1}
I20260620 06:09:24.830498 26048 raft_consensus.cc:348] T e8daa5c69e82442b9e436139959f91b9 P d7f871fdb6814adabd0538690e2a65a3 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d1347dff7a94072ad8cbe2deef6f05a" member_type: VOTER last_known_addr { host: "127.11.102.2" port: 36731 } } peers { permanent_uuid: "d7f871fdb6814adabd0538690e2a65a3" member_type: VOTER last_known_addr { host: "127.11.102.1" port: 41679 } } peers { permanent_uuid: "7c99aa7fa7834c89968adb1265fa3603" member_type: VOTER last_known_addr { host: "127.11.102.3" port: 43859 } }
I20260620 06:09:24.831029 26048 raft_consensus.cc:729] T e8daa5c69e82442b9e436139959f91b9 P d7f871fdb6814adabd0538690e2a65a3 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: d7f871fdb6814adabd0538690e2a65a3, State: Initialized, Role: FOLLOWER
I20260620 06:09:24.831205 26156 heartbeater.cc:347] Connected to a master server at 127.11.102.62:41475
I20260620 06:09:24.831408 26156 heartbeater.cc:464] Registering TS with master...
I20260620 06:09:24.831548 26048 consensus_queue.cc:260] T e8daa5c69e82442b9e436139959f91b9 P d7f871fdb6814adabd0538690e2a65a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d1347dff7a94072ad8cbe2deef6f05a" member_type: VOTER last_known_addr { host: "127.11.102.2" port: 36731 } } peers { permanent_uuid: "d7f871fdb6814adabd0538690e2a65a3" member_type: VOTER last_known_addr { host: "127.11.102.1" port: 41679 } } peers { permanent_uuid: "7c99aa7fa7834c89968adb1265fa3603" member_type: VOTER last_known_addr { host: "127.11.102.3" port: 43859 } }
I20260620 06:09:24.831948 26156 heartbeater.cc:511] Master 127.11.102.62:41475 requested a full tablet report, sending...
I20260620 06:09:24.832537 26048 ts_tablet_manager.cc:1428] T e8daa5c69e82442b9e436139959f91b9 P d7f871fdb6814adabd0538690e2a65a3: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20260620 06:09:24.834473 25966 ts_manager.cc:194] Registered new tserver with Master: d7f871fdb6814adabd0538690e2a65a3 (127.11.102.1:41679)
I20260620 06:09:24.838344 25966 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.102.1@KRBTEST.COM'} at 127.11.102.1:33629
I20260620 06:09:24.840687 26156 heartbeater.cc:503] Master 127.11.102.62:41475 was elected leader, sending a full tablet report...
Jun 20 06:09:24 dist-test-slave-9hm5 krb5kdc[25389](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935764, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.102.62@KRBTEST.COM for kudu/127.11.102.1@KRBTEST.COM
I20260620 06:09:24.848294 26163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:09:24.837703 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.102.1:41679 (local address 127.0.0.1:50764)
0620 06:09:24.838100 (+ 397us) negotiation.cc:107] Waiting for socket to connect
0620 06:09:24.838141 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0620 06:09:24.838271 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:09:24.838943 (+ 672us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:09:24.838949 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:09:24.839359 (+ 410us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:09:24.839517 (+ 158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:09:24.839525 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:09:24.840967 (+ 1442us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:09:24.840970 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:09:24.841731 (+ 761us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:09:24.841740 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:09:24.842334 (+ 594us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:09:24.842338 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:09:24.842464 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:09:24.843050 (+ 586us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:09:24.843062 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:09:24.844495 (+ 1433us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:09:24.846472 (+ 1977us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:09:24.846481 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:09:24.846486 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:09:24.846718 (+ 232us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:09:24.847044 (+ 326us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:09:24.847047 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:09:24.847049 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:09:24.847138 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:09:24.847632 (+ 494us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:09:24.847638 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:09:24.847763 (+ 125us) client_negotiation.cc:769] Sending connection context
0620 06:09:24.847913 (+ 150us) client_negotiation.cc:240] Negotiation successful
0620 06:09:24.848105 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":210,"thread_start_us":130,"threads_started":1}
I20260620 06:09:24.848300 26164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:09:24.837755 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50764 (local address 127.11.102.1:41679)
0620 06:09:24.838539 (+ 784us) server_negotiation.cc:206] Beginning negotiation
0620 06:09:24.838552 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 06:09:24.838573 (+ 21us) server_negotiation.cc:407] Connection header received
0620 06:09:24.838633 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:09:24.838636 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:09:24.838790 (+ 154us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:09:24.838941 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:09:24.839666 (+ 725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:09:24.840842 (+ 1176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:09:24.841863 (+ 1021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:09:24.842204 (+ 341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:09:24.842333 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:09:24.844643 (+ 2310us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:09:24.844660 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:09:24.844670 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:09:24.844691 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:09:24.846332 (+ 1641us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:09:24.846843 (+ 511us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:09:24.846848 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:09:24.846850 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:09:24.846934 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:09:24.847258 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:09:24.847262 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:09:24.847263 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:09:24.847487 (+ 224us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:09:24.847655 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0620 06:09:24.848007 (+ 352us) server_negotiation.cc:299] Negotiation successful
0620 06:09:24.848163 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":651,"thread_start_us":94,"threads_started":1}
I20260620 06:09:24.849291 26090 tablet_service.cc:1514] Processing DeleteTablet for tablet e8daa5c69e82442b9e436139959f91b9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 06:09:24 UTC) from {username='kudu', principal='kudu/127.11.102.62@KRBTEST.COM'} at 127.0.0.1:50764
I20260620 06:09:24.850149 26165 tablet_replica.cc:330] stopping tablet replica
I20260620 06:09:24.850461 26165 raft_consensus.cc:2229] T e8daa5c69e82442b9e436139959f91b9 P d7f871fdb6814adabd0538690e2a65a3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 06:09:24.850718 26165 raft_consensus.cc:2258] T e8daa5c69e82442b9e436139959f91b9 P d7f871fdb6814adabd0538690e2a65a3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 06:09:24.851840 26165 ts_tablet_manager.cc:1905] T e8daa5c69e82442b9e436139959f91b9 P d7f871fdb6814adabd0538690e2a65a3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 06:09:24.856081 26165 ts_tablet_manager.cc:1918] T e8daa5c69e82442b9e436139959f91b9 P d7f871fdb6814adabd0538690e2a65a3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 06:09:24.856184 26165 log.cc:1196] T e8daa5c69e82442b9e436139959f91b9 P d7f871fdb6814adabd0538690e2a65a3: Deleting WAL directory at /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935603120653-11672-0/minicluster-data/ts-0/wal/wals/e8daa5c69e82442b9e436139959f91b9
I20260620 06:09:24.856459 26165 ts_tablet_manager.cc:1939] T e8daa5c69e82442b9e436139959f91b9 P d7f871fdb6814adabd0538690e2a65a3: Deleting consensus metadata
I20260620 06:09:24.858376 25952 catalog_manager.cc:4841] TS d7f871fdb6814adabd0538690e2a65a3 (127.11.102.1:41679): tablet e8daa5c69e82442b9e436139959f91b9 (table test-table [id=d69129a39f9f4ee089af9b6114498a2f]) successfully deleted
W20260620 06:09:24.938400 26161 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:09:24.938709 26161 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:09:24.938784 26161 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:09:24.942824 26161 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:09:24.942917 26161 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:09:24.943006 26161 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.102.2
I20260620 06:09:24.948268 26161 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.102.20:37051
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935603120653-11672-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935603120653-11672-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935603120653-11672-0/krb5kdc/kudu_127.11.102.2.keytab
--principal=kudu/127.11.102.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.102.2:36731
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935603120653-11672-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.11.102.2
--webserver_port=42277
--tserver_master_addrs=127.11.102.62:41475
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26161
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.102.2
--log_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935603120653-11672-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 05:28:48 UTC on ccycloud.quasar-fixhms.root.comops.site
build id 4457020
I20260620 06:09:24.949361 26161 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:09:24.950421 26161 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 06:09:24.960364 26170 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:09:24.960368 26171 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:09:24.960551 26161 server_base.cc:1034] running on GCE node
W20260620 06:09:24.960395 26173 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:09:24.961205 26161 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:09:24.961865 26161 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:09:24.963061 26161 hybrid_clock.cc:648] HybridClock initialized: now 1781935764963042 us; error 34 us; skew 500 ppm
Jun 20 06:09:24 dist-test-slave-9hm5 krb5kdc[25389](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935764, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.102.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 06:09:24.966042 26161 init.cc:377] Logged in from keytab as kudu/127.11.102.2@KRBTEST.COM (short username kudu)
I20260620 06:09:24.966915 26161 webserver.cc:460] Webserver started at http://127.11.102.2:42277/ using document root <none> and password file <none>
I20260620 06:09:24.967499 26161 fs_manager.cc:359] Metadata directory not provided
I20260620 06:09:24.967614 26161 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:09:24.971434 26161 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.001s
I20260620 06:09:24.973591 26180 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:09:24.974915 26161 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 06:09:24.975059 26161 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935603120653-11672-0/minicluster-data/ts-1/data,/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935603120653-11672-0/minicluster-data/ts-1/wal
uuid: "0d1347dff7a94072ad8cbe2deef6f05a"
format_stamp: "Formatted at 2026-06-20 06:09:23 on dist-test-slave-9hm5"
server_key: "6dd3b8101a03a814b1a68fe753780af8"
server_key_iv: "9bcd1732e4c77ea13bb2b9fadd277fca"
server_key_version: "encryptionkey@0"
I20260620 06:09:24.975669 26161 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935603120653-11672-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935603120653-11672-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935603120653-11672-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:09:24.993340 26161 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:09:24.996824 26161 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:09:24.997044 26161 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:09:24.997668 26161 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 06:09:24.998539 26161 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 06:09:24.998618 26161 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 06:09:24.998688 26161 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 06:09:24.998745 26161 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 06:09:25.007772 26161 rpc_server.cc:307] RPC server started. Bound to: 127.11.102.2:36731
I20260620 06:09:25.007827 26293 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.102.2:36731 every 8 connection(s)
I20260620 06:09:25.008811 26161 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935603120653-11672-0/minicluster-data/ts-1/data/info.pb
I20260620 06:09:25.016254 11672 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9HMp43/build/debug/bin/kudu as pid 26161
I20260620 06:09:25.016682 11672 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9HMp43/build/debug/bin/kudu
/tmp/dist-test-task9HMp43/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935603120653-11672-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935603120653-11672-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935603120653-11672-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935603120653-11672-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.102.3:43859
--local_ip_for_outbound_sockets=127.11.102.3
--tserver_master_addrs=127.11.102.62:41475
--webserver_port=33315
--webserver_interface=127.11.102.3
--keytab_file=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935603120653-11672-0/krb5kdc/kudu_127.11.102.3.keytab
--principal=kudu/127.11.102.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.102.20:37051
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935603120653-11672-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935603120653-11672-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935603120653-11672-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 06:09:25 dist-test-slave-9hm5 krb5kdc[25389](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935764, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.102.2@KRBTEST.COM for kudu/127.11.102.62@KRBTEST.COM
I20260620 06:09:25.023132 26158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:09:25.010642 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.102.2:47521 (local address 127.11.102.62:41475)
0620 06:09:25.010789 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0620 06:09:25.010792 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 06:09:25.011647 (+ 855us) server_negotiation.cc:407] Connection header received
0620 06:09:25.013072 (+ 1425us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:09:25.013076 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:09:25.013140 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:09:25.013252 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:09:25.014480 (+ 1228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:09:25.015012 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:09:25.016239 (+ 1227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:09:25.016534 (+ 295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:09:25.016677 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:09:25.019388 (+ 2711us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:09:25.019391 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:09:25.019393 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:09:25.019417 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:09:25.021023 (+ 1606us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:09:25.021547 (+ 524us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:09:25.021552 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:09:25.021554 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:09:25.021634 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:09:25.021985 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:09:25.021990 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:09:25.021991 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:09:25.022186 (+ 195us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:09:25.022308 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0620 06:09:25.022824 (+ 516us) server_negotiation.cc:299] Negotiation successful
0620 06:09:25.022962 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260620 06:09:25.024041 26296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:09:25.010967 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.102.62:41475 (local address 127.11.102.2:47521)
0620 06:09:25.011441 (+ 474us) negotiation.cc:107] Waiting for socket to connect
0620 06:09:25.011483 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0620 06:09:25.012840 (+ 1357us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:09:25.013437 (+ 597us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:09:25.013444 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:09:25.013820 (+ 376us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:09:25.014309 (+ 489us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:09:25.014322 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:09:25.015147 (+ 825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:09:25.015151 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:09:25.016092 (+ 941us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:09:25.016101 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:09:25.016722 (+ 621us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:09:25.016727 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:09:25.016941 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:09:25.017566 (+ 625us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:09:25.017582 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:09:25.019207 (+ 1625us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:09:25.021149 (+ 1942us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:09:25.021155 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:09:25.021172 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:09:25.021436 (+ 264us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:09:25.021736 (+ 300us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:09:25.021740 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:09:25.021743 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:09:25.021885 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:09:25.022316 (+ 431us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:09:25.022325 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:09:25.022580 (+ 255us) client_negotiation.cc:769] Sending connection context
0620 06:09:25.022786 (+ 206us) client_negotiation.cc:240] Negotiation successful
0620 06:09:25.023110 (+ 324us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":280,"thread_start_us":105,"threads_started":1}
I20260620 06:09:25.025269 26294 heartbeater.cc:347] Connected to a master server at 127.11.102.62:41475
I20260620 06:09:25.025560 26294 heartbeater.cc:464] Registering TS with master...
I20260620 06:09:25.026211 26294 heartbeater.cc:511] Master 127.11.102.62:41475 requested a full tablet report, sending...
I20260620 06:09:25.027436 25966 ts_manager.cc:194] Registered new tserver with Master: 0d1347dff7a94072ad8cbe2deef6f05a (127.11.102.2:36731)
I20260620 06:09:25.028293 25966 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.102.2@KRBTEST.COM'} at 127.11.102.2:47521
W20260620 06:09:25.130900 26297 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:09:25.131161 26297 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:09:25.131204 26297 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:09:25.135250 26297 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:09:25.135308 26297 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:09:25.135365 26297 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.102.3
I20260620 06:09:25.140470 26297 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.102.20:37051
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935603120653-11672-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935603120653-11672-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935603120653-11672-0/krb5kdc/kudu_127.11.102.3.keytab
--principal=kudu/127.11.102.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.102.3:43859
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935603120653-11672-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.11.102.3
--webserver_port=33315
--tserver_master_addrs=127.11.102.62:41475
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26297
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.102.3
--log_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935603120653-11672-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 05:28:48 UTC on ccycloud.quasar-fixhms.root.comops.site
build id 4457020
I20260620 06:09:25.141417 26297 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:09:25.142365 26297 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 06:09:25.152087 26302 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:09:25.152087 26303 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:09:25.152216 26297 server_base.cc:1034] running on GCE node
W20260620 06:09:25.152155 26305 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:09:25.152720 26297 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:09:25.153475 26297 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:09:25.154650 26297 hybrid_clock.cc:648] HybridClock initialized: now 1781935765154633 us; error 32 us; skew 500 ppm
Jun 20 06:09:25 dist-test-slave-9hm5 krb5kdc[25389](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935765, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.102.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 06:09:25.157851 26297 init.cc:377] Logged in from keytab as kudu/127.11.102.3@KRBTEST.COM (short username kudu)
I20260620 06:09:25.158732 26297 webserver.cc:460] Webserver started at http://127.11.102.3:33315/ using document root <none> and password file <none>
I20260620 06:09:25.159231 26297 fs_manager.cc:359] Metadata directory not provided
I20260620 06:09:25.159299 26297 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:09:25.163079 26297 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260620 06:09:25.165279 26312 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:09:25.166512 26297 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260620 06:09:25.166633 26297 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935603120653-11672-0/minicluster-data/ts-2/data,/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935603120653-11672-0/minicluster-data/ts-2/wal
uuid: "7c99aa7fa7834c89968adb1265fa3603"
format_stamp: "Formatted at 2026-06-20 06:09:24 on dist-test-slave-9hm5"
server_key: "b3b0ed89c1cc4c95c4ba805a87b5e8c5"
server_key_iv: "57f8a6a192208426f9de3dc1b98d0013"
server_key_version: "encryptionkey@0"
I20260620 06:09:25.167176 26297 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935603120653-11672-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935603120653-11672-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935603120653-11672-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:09:25.200029 26297 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:09:25.203442 26297 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:09:25.203675 26297 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:09:25.204316 26297 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 06:09:25.205119 26297 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 06:09:25.205165 26297 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 06:09:25.205237 26297 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 06:09:25.205271 26297 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 06:09:25.214573 26297 rpc_server.cc:307] RPC server started. Bound to: 127.11.102.3:43859
I20260620 06:09:25.214597 26425 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.102.3:43859 every 8 connection(s)
I20260620 06:09:25.215662 26297 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935603120653-11672-0/minicluster-data/ts-2/data/info.pb
I20260620 06:09:25.224439 11672 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9HMp43/build/debug/bin/kudu as pid 26297
Jun 20 06:09:25 dist-test-slave-9hm5 krb5kdc[25389](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935765, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.102.3@KRBTEST.COM for kudu/127.11.102.62@KRBTEST.COM
I20260620 06:09:25.230202 26158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:09:25.217575 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.102.3:35151 (local address 127.11.102.62:41475)
0620 06:09:25.217727 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0620 06:09:25.217731 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 06:09:25.218602 (+ 871us) server_negotiation.cc:407] Connection header received
0620 06:09:25.220095 (+ 1493us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:09:25.220099 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:09:25.220169 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:09:25.220285 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:09:25.221483 (+ 1198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:09:25.222016 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:09:25.223217 (+ 1201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:09:25.223457 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:09:25.223583 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:09:25.226484 (+ 2901us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:09:25.226488 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:09:25.226493 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:09:25.226523 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:09:25.228199 (+ 1676us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:09:25.228697 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:09:25.228700 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:09:25.228702 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:09:25.228797 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:09:25.229146 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:09:25.229151 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:09:25.229152 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:09:25.229425 (+ 273us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:09:25.229522 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0620 06:09:25.229977 (+ 455us) server_negotiation.cc:299] Negotiation successful
0620 06:09:25.230072 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260620 06:09:25.231025 26428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:09:25.217970 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.102.62:41475 (local address 127.11.102.3:35151)
0620 06:09:25.218428 (+ 458us) negotiation.cc:107] Waiting for socket to connect
0620 06:09:25.218471 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0620 06:09:25.219869 (+ 1398us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:09:25.220464 (+ 595us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:09:25.220470 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:09:25.220843 (+ 373us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:09:25.221319 (+ 476us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:09:25.221332 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:09:25.222167 (+ 835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:09:25.222171 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:09:25.223072 (+ 901us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:09:25.223081 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:09:25.223663 (+ 582us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:09:25.223669 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:09:25.223832 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:09:25.224403 (+ 571us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:09:25.224426 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:09:25.226309 (+ 1883us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:09:25.228312 (+ 2003us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:09:25.228316 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:09:25.228328 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:09:25.228570 (+ 242us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:09:25.228904 (+ 334us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:09:25.228907 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:09:25.228909 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:09:25.229043 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:09:25.229525 (+ 482us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:09:25.229531 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:09:25.229764 (+ 233us) client_negotiation.cc:769] Sending connection context
0620 06:09:25.229925 (+ 161us) client_negotiation.cc:240] Negotiation successful
0620 06:09:25.230187 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":262,"thread_start_us":108,"threads_started":1}
I20260620 06:09:25.232576 26426 heartbeater.cc:347] Connected to a master server at 127.11.102.62:41475
I20260620 06:09:25.232872 26426 heartbeater.cc:464] Registering TS with master...
I20260620 06:09:25.233539 26426 heartbeater.cc:511] Master 127.11.102.62:41475 requested a full tablet report, sending...
I20260620 06:09:25.234704 25966 ts_manager.cc:194] Registered new tserver with Master: 7c99aa7fa7834c89968adb1265fa3603 (127.11.102.3:43859)
I20260620 06:09:25.234890 26430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:09:25.225191 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48658 (local address 127.11.102.62:41475)
0620 06:09:25.225447 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0620 06:09:25.225451 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 06:09:25.225590 (+ 139us) server_negotiation.cc:407] Connection header received
0620 06:09:25.225729 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:09:25.225731 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:09:25.225769 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:09:25.225847 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:09:25.226665 (+ 818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:09:25.227411 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:09:25.228615 (+ 1204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:09:25.228848 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:09:25.228963 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:09:25.231284 (+ 2321us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:09:25.231288 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:09:25.231290 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:09:25.231317 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:09:25.232972 (+ 1655us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:09:25.233505 (+ 533us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:09:25.233508 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:09:25.233509 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:09:25.233636 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:09:25.233977 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:09:25.233982 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:09:25.233984 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:09:25.234191 (+ 207us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:09:25.234383 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0620 06:09:25.234604 (+ 221us) server_negotiation.cc:299] Negotiation successful
0620 06:09:25.234753 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":188,"thread_start_us":89,"threads_started":1}
I20260620 06:09:25.235419 25966 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.102.3@KRBTEST.COM'} at 127.11.102.3:35151
I20260620 06:09:25.236680 11672 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 06:09:25.240756 26430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:09:25.237320 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48664 (local address 127.11.102.62:41475)
0620 06:09:25.237431 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0620 06:09:25.237435 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 06:09:25.237593 (+ 158us) server_negotiation.cc:407] Connection header received
0620 06:09:25.237713 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:09:25.237715 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:09:25.237795 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:09:25.237874 (+ 79us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 06:09:25.238102 (+ 228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:09:25.238589 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:09:25.239378 (+ 789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:09:25.239587 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:09:25.239734 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:09:25.240050 (+ 316us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 06:09:25.240393 (+ 343us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 06:09:25.240528 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0620 06:09:25.240588 (+ 60us) server_negotiation.cc:299] Negotiation successful
0620 06:09:25.240640 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260620 06:09:25.241335 25966 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:48664:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 06:09:25.244459 25966 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 06:09:25.252451 26090 tablet_service.cc:1467] Processing CreateTablet for tablet b729a999fd6b46d1af62153d893b6656 (DEFAULT_TABLE table=test-table [id=bc583ad1632a474d89a40ad17502dc09]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 06:09:25.253094 26090 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b729a999fd6b46d1af62153d893b6656. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 06:09:25.256134 26048 tablet_bootstrap.cc:492] T b729a999fd6b46d1af62153d893b6656 P d7f871fdb6814adabd0538690e2a65a3: Bootstrap starting.
I20260620 06:09:25.256594 26163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:09:25.250904 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.102.3:43859 (local address 127.0.0.1:59988)
0620 06:09:25.251065 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0620 06:09:25.251079 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 06:09:25.251158 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:09:25.252214 (+ 1056us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:09:25.252217 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:09:25.252235 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 06:09:25.252332 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:09:25.252338 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:09:25.253684 (+ 1346us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:09:25.253687 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:09:25.254937 (+ 1250us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:09:25.254947 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:09:25.256144 (+ 1197us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:09:25.256151 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:09:25.256289 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:09:25.256348 (+ 59us) client_negotiation.cc:769] Sending connection context
0620 06:09:25.256406 (+ 58us) client_negotiation.cc:240] Negotiation successful
0620 06:09:25.256497 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":93}
I20260620 06:09:25.256763 26436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:09:25.251280 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.102.2:36731 (local address 127.0.0.1:40030)
0620 06:09:25.251522 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0620 06:09:25.251537 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 06:09:25.251652 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:09:25.252654 (+ 1002us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:09:25.252659 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:09:25.252678 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 06:09:25.252761 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:09:25.252768 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:09:25.253866 (+ 1098us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:09:25.253869 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:09:25.255188 (+ 1319us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:09:25.255197 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:09:25.256291 (+ 1094us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:09:25.256295 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:09:25.256386 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:09:25.256410 (+ 24us) client_negotiation.cc:769] Sending connection context
0620 06:09:25.256475 (+ 65us) client_negotiation.cc:240] Negotiation successful
0620 06:09:25.256550 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":91,"threads_started":1}
I20260620 06:09:25.257033 26048 tablet_bootstrap.cc:654] T b729a999fd6b46d1af62153d893b6656 P d7f871fdb6814adabd0538690e2a65a3: Neither blocks nor log segments found. Creating new log.
I20260620 06:09:25.257102 26437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:09:25.251465 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40030 (local address 127.11.102.2:36731)
0620 06:09:25.252018 (+ 553us) server_negotiation.cc:206] Beginning negotiation
0620 06:09:25.252030 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 06:09:25.252051 (+ 21us) server_negotiation.cc:407] Connection header received
0620 06:09:25.252122 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:09:25.252125 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:09:25.252342 (+ 217us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:09:25.252487 (+ 145us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 06:09:25.252868 (+ 381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:09:25.253719 (+ 851us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:09:25.255303 (+ 1584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:09:25.256058 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:09:25.256177 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:09:25.256769 (+ 592us) server_negotiation.cc:1035] Waiting for connection context
0620 06:09:25.256826 (+ 57us) server_negotiation.cc:299] Negotiation successful
0620 06:09:25.256930 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":443,"thread_start_us":112,"threads_started":1}
I20260620 06:09:25.257474 26435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:09:25.251057 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59988 (local address 127.11.102.3:43859)
0620 06:09:25.251676 (+ 619us) server_negotiation.cc:206] Beginning negotiation
0620 06:09:25.251691 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 06:09:25.251717 (+ 26us) server_negotiation.cc:407] Connection header received
0620 06:09:25.251800 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:09:25.251804 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:09:25.252051 (+ 247us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:09:25.252225 (+ 174us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 06:09:25.252472 (+ 247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:09:25.253539 (+ 1067us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:09:25.255068 (+ 1529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:09:25.256000 (+ 932us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:09:25.256841 (+ 841us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:09:25.257151 (+ 310us) server_negotiation.cc:1035] Waiting for connection context
0620 06:09:25.257213 (+ 62us) server_negotiation.cc:299] Negotiation successful
0620 06:09:25.257307 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":489,"thread_start_us":154,"threads_started":1}
I20260620 06:09:25.258494 26048 tablet_bootstrap.cc:492] T b729a999fd6b46d1af62153d893b6656 P d7f871fdb6814adabd0538690e2a65a3: No bootstrap required, opened a new log
I20260620 06:09:25.258589 26048 ts_tablet_manager.cc:1397] T b729a999fd6b46d1af62153d893b6656 P d7f871fdb6814adabd0538690e2a65a3: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.001s
I20260620 06:09:25.258989 26048 raft_consensus.cc:348] T b729a999fd6b46d1af62153d893b6656 P d7f871fdb6814adabd0538690e2a65a3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c99aa7fa7834c89968adb1265fa3603" member_type: VOTER last_known_addr { host: "127.11.102.3" port: 43859 } } peers { permanent_uuid: "0d1347dff7a94072ad8cbe2deef6f05a" member_type: VOTER last_known_addr { host: "127.11.102.2" port: 36731 } } peers { permanent_uuid: "d7f871fdb6814adabd0538690e2a65a3" member_type: VOTER last_known_addr { host: "127.11.102.1" port: 41679 } }
I20260620 06:09:25.259133 26048 raft_consensus.cc:374] T b729a999fd6b46d1af62153d893b6656 P d7f871fdb6814adabd0538690e2a65a3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 06:09:25.259181 26048 raft_consensus.cc:729] T b729a999fd6b46d1af62153d893b6656 P d7f871fdb6814adabd0538690e2a65a3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d7f871fdb6814adabd0538690e2a65a3, State: Initialized, Role: FOLLOWER
I20260620 06:09:25.259308 26048 consensus_queue.cc:260] T b729a999fd6b46d1af62153d893b6656 P d7f871fdb6814adabd0538690e2a65a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c99aa7fa7834c89968adb1265fa3603" member_type: VOTER last_known_addr { host: "127.11.102.3" port: 43859 } } peers { permanent_uuid: "0d1347dff7a94072ad8cbe2deef6f05a" member_type: VOTER last_known_addr { host: "127.11.102.2" port: 36731 } } peers { permanent_uuid: "d7f871fdb6814adabd0538690e2a65a3" member_type: VOTER last_known_addr { host: "127.11.102.1" port: 41679 } }
I20260620 06:09:25.259562 26048 ts_tablet_manager.cc:1428] T b729a999fd6b46d1af62153d893b6656 P d7f871fdb6814adabd0538690e2a65a3: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260620 06:09:25.260591 26228 tablet_service.cc:1467] Processing CreateTablet for tablet b729a999fd6b46d1af62153d893b6656 (DEFAULT_TABLE table=test-table [id=bc583ad1632a474d89a40ad17502dc09]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 06:09:25.261152 26360 tablet_service.cc:1467] Processing CreateTablet for tablet b729a999fd6b46d1af62153d893b6656 (DEFAULT_TABLE table=test-table [id=bc583ad1632a474d89a40ad17502dc09]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 06:09:25.261793 26228 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b729a999fd6b46d1af62153d893b6656. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 06:09:25.262190 26360 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b729a999fd6b46d1af62153d893b6656. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 06:09:25.268678 26439 tablet_bootstrap.cc:492] T b729a999fd6b46d1af62153d893b6656 P 0d1347dff7a94072ad8cbe2deef6f05a: Bootstrap starting.
I20260620 06:09:25.268687 26440 tablet_bootstrap.cc:492] T b729a999fd6b46d1af62153d893b6656 P 7c99aa7fa7834c89968adb1265fa3603: Bootstrap starting.
I20260620 06:09:25.270905 26440 tablet_bootstrap.cc:654] T b729a999fd6b46d1af62153d893b6656 P 7c99aa7fa7834c89968adb1265fa3603: Neither blocks nor log segments found. Creating new log.
I20260620 06:09:25.271447 26439 tablet_bootstrap.cc:654] T b729a999fd6b46d1af62153d893b6656 P 0d1347dff7a94072ad8cbe2deef6f05a: Neither blocks nor log segments found. Creating new log.
I20260620 06:09:25.272171 26440 log.cc:824] T b729a999fd6b46d1af62153d893b6656 P 7c99aa7fa7834c89968adb1265fa3603: Log is configured to *not* fsync() on all Append() calls
I20260620 06:09:25.272228 26439 log.cc:824] T b729a999fd6b46d1af62153d893b6656 P 0d1347dff7a94072ad8cbe2deef6f05a: Log is configured to *not* fsync() on all Append() calls
I20260620 06:09:25.273928 26439 tablet_bootstrap.cc:492] T b729a999fd6b46d1af62153d893b6656 P 0d1347dff7a94072ad8cbe2deef6f05a: No bootstrap required, opened a new log
I20260620 06:09:25.273938 26440 tablet_bootstrap.cc:492] T b729a999fd6b46d1af62153d893b6656 P 7c99aa7fa7834c89968adb1265fa3603: No bootstrap required, opened a new log
I20260620 06:09:25.274134 26440 ts_tablet_manager.cc:1397] T b729a999fd6b46d1af62153d893b6656 P 7c99aa7fa7834c89968adb1265fa3603: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.004s
I20260620 06:09:25.274134 26439 ts_tablet_manager.cc:1397] T b729a999fd6b46d1af62153d893b6656 P 0d1347dff7a94072ad8cbe2deef6f05a: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260620 06:09:25.277460 26439 raft_consensus.cc:348] T b729a999fd6b46d1af62153d893b6656 P 0d1347dff7a94072ad8cbe2deef6f05a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c99aa7fa7834c89968adb1265fa3603" member_type: VOTER last_known_addr { host: "127.11.102.3" port: 43859 } } peers { permanent_uuid: "0d1347dff7a94072ad8cbe2deef6f05a" member_type: VOTER last_known_addr { host: "127.11.102.2" port: 36731 } } peers { permanent_uuid: "d7f871fdb6814adabd0538690e2a65a3" member_type: VOTER last_known_addr { host: "127.11.102.1" port: 41679 } }
I20260620 06:09:25.277495 26440 raft_consensus.cc:348] T b729a999fd6b46d1af62153d893b6656 P 7c99aa7fa7834c89968adb1265fa3603 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c99aa7fa7834c89968adb1265fa3603" member_type: VOTER last_known_addr { host: "127.11.102.3" port: 43859 } } peers { permanent_uuid: "0d1347dff7a94072ad8cbe2deef6f05a" member_type: VOTER last_known_addr { host: "127.11.102.2" port: 36731 } } peers { permanent_uuid: "d7f871fdb6814adabd0538690e2a65a3" member_type: VOTER last_known_addr { host: "127.11.102.1" port: 41679 } }
I20260620 06:09:25.277707 26439 raft_consensus.cc:374] T b729a999fd6b46d1af62153d893b6656 P 0d1347dff7a94072ad8cbe2deef6f05a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 06:09:25.277741 26440 raft_consensus.cc:374] T b729a999fd6b46d1af62153d893b6656 P 7c99aa7fa7834c89968adb1265fa3603 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 06:09:25.277743 26439 raft_consensus.cc:729] T b729a999fd6b46d1af62153d893b6656 P 0d1347dff7a94072ad8cbe2deef6f05a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d1347dff7a94072ad8cbe2deef6f05a, State: Initialized, Role: FOLLOWER
I20260620 06:09:25.277805 26440 raft_consensus.cc:729] T b729a999fd6b46d1af62153d893b6656 P 7c99aa7fa7834c89968adb1265fa3603 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7c99aa7fa7834c89968adb1265fa3603, State: Initialized, Role: FOLLOWER
I20260620 06:09:25.278208 26439 consensus_queue.cc:260] T b729a999fd6b46d1af62153d893b6656 P 0d1347dff7a94072ad8cbe2deef6f05a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c99aa7fa7834c89968adb1265fa3603" member_type: VOTER last_known_addr { host: "127.11.102.3" port: 43859 } } peers { permanent_uuid: "0d1347dff7a94072ad8cbe2deef6f05a" member_type: VOTER last_known_addr { host: "127.11.102.2" port: 36731 } } peers { permanent_uuid: "d7f871fdb6814adabd0538690e2a65a3" member_type: VOTER last_known_addr { host: "127.11.102.1" port: 41679 } }
I20260620 06:09:25.278208 26440 consensus_queue.cc:260] T b729a999fd6b46d1af62153d893b6656 P 7c99aa7fa7834c89968adb1265fa3603 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c99aa7fa7834c89968adb1265fa3603" member_type: VOTER last_known_addr { host: "127.11.102.3" port: 43859 } } peers { permanent_uuid: "0d1347dff7a94072ad8cbe2deef6f05a" member_type: VOTER last_known_addr { host: "127.11.102.2" port: 36731 } } peers { permanent_uuid: "d7f871fdb6814adabd0538690e2a65a3" member_type: VOTER last_known_addr { host: "127.11.102.1" port: 41679 } }
I20260620 06:09:25.278898 26426 heartbeater.cc:503] Master 127.11.102.62:41475 was elected leader, sending a full tablet report...
I20260620 06:09:25.278914 26294 heartbeater.cc:503] Master 127.11.102.62:41475 was elected leader, sending a full tablet report...
I20260620 06:09:25.279261 26439 ts_tablet_manager.cc:1428] T b729a999fd6b46d1af62153d893b6656 P 0d1347dff7a94072ad8cbe2deef6f05a: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260620 06:09:25.280319 26440 ts_tablet_manager.cc:1428] T b729a999fd6b46d1af62153d893b6656 P 7c99aa7fa7834c89968adb1265fa3603: Time spent starting tablet: real 0.006s user 0.005s sys 0.001s
I20260620 06:09:25.309512 26444 raft_consensus.cc:482] T b729a999fd6b46d1af62153d893b6656 P 7c99aa7fa7834c89968adb1265fa3603 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 06:09:25.309799 26444 raft_consensus.cc:504] T b729a999fd6b46d1af62153d893b6656 P 7c99aa7fa7834c89968adb1265fa3603 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c99aa7fa7834c89968adb1265fa3603" member_type: VOTER last_known_addr { host: "127.11.102.3" port: 43859 } } peers { permanent_uuid: "0d1347dff7a94072ad8cbe2deef6f05a" member_type: VOTER last_known_addr { host: "127.11.102.2" port: 36731 } } peers { permanent_uuid: "d7f871fdb6814adabd0538690e2a65a3" member_type: VOTER last_known_addr { host: "127.11.102.1" port: 41679 } }
W20260620 06:09:25.310180 26157 tablet.cc:2369] T b729a999fd6b46d1af62153d893b6656 P d7f871fdb6814adabd0538690e2a65a3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 06:09:25.311102 26444 leader_election.cc:288] T b729a999fd6b46d1af62153d893b6656 P 7c99aa7fa7834c89968adb1265fa3603 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0d1347dff7a94072ad8cbe2deef6f05a (127.11.102.2:36731), d7f871fdb6814adabd0538690e2a65a3 (127.11.102.1:41679)
I20260620 06:09:25.315642 26164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:09:25.311532 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.102.3:55561 (local address 127.11.102.1:41679)
0620 06:09:25.311752 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0620 06:09:25.311757 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 06:09:25.311988 (+ 231us) server_negotiation.cc:407] Connection header received
0620 06:09:25.312043 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:09:25.312045 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:09:25.312092 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:09:25.312204 (+ 112us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 06:09:25.312460 (+ 256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:09:25.313193 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:09:25.314576 (+ 1383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:09:25.315034 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:09:25.315176 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:09:25.315326 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0620 06:09:25.315407 (+ 81us) server_negotiation.cc:299] Negotiation successful
0620 06:09:25.315485 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":132}
I20260620 06:09:25.315662 26437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:09:25.311449 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.102.3:55185 (local address 127.11.102.2:36731)
0620 06:09:25.311584 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0620 06:09:25.311591 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 06:09:25.311681 (+ 90us) server_negotiation.cc:407] Connection header received
0620 06:09:25.311744 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:09:25.311747 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:09:25.311809 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:09:25.311873 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 06:09:25.312226 (+ 353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:09:25.312994 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:09:25.314618 (+ 1624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:09:25.315028 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:09:25.315171 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:09:25.315224 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0620 06:09:25.315442 (+ 218us) server_negotiation.cc:299] Negotiation successful
0620 06:09:25.315515 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260620 06:09:25.315660 26428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:09:25.311336 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.102.2:36731 (local address 127.11.102.3:55185)
0620 06:09:25.311471 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0620 06:09:25.311486 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 06:09:25.311578 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:09:25.311929 (+ 351us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:09:25.311933 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:09:25.311958 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 06:09:25.312101 (+ 143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:09:25.312107 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:09:25.313106 (+ 999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:09:25.313109 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:09:25.314442 (+ 1333us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:09:25.314453 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:09:25.315169 (+ 716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:09:25.315173 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:09:25.315296 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:09:25.315317 (+ 21us) client_negotiation.cc:769] Sending connection context
0620 06:09:25.315420 (+ 103us) client_negotiation.cc:240] Negotiation successful
0620 06:09:25.315496 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":45,"mutex_wait_us":69}
I20260620 06:09:25.315673 26445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:09:25.311345 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.102.1:41679 (local address 127.11.102.3:55561)
0620 06:09:25.311844 (+ 499us) negotiation.cc:107] Waiting for socket to connect
0620 06:09:25.311883 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0620 06:09:25.311971 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:09:25.312236 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:09:25.312239 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:09:25.312259 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 06:09:25.312347 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:09:25.312351 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:09:25.313299 (+ 948us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:09:25.313302 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:09:25.314444 (+ 1142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:09:25.314453 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:09:25.315171 (+ 718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:09:25.315175 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:09:25.315292 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:09:25.315318 (+ 26us) client_negotiation.cc:769] Sending connection context
0620 06:09:25.315395 (+ 77us) client_negotiation.cc:240] Negotiation successful
0620 06:09:25.315473 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":441,"mutex_wait_us":110,"thread_start_us":154,"threads_started":1}
I20260620 06:09:25.316412 26110 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b729a999fd6b46d1af62153d893b6656" candidate_uuid: "7c99aa7fa7834c89968adb1265fa3603" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d7f871fdb6814adabd0538690e2a65a3" is_pre_election: true
I20260620 06:09:25.316428 26248 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b729a999fd6b46d1af62153d893b6656" candidate_uuid: "7c99aa7fa7834c89968adb1265fa3603" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d1347dff7a94072ad8cbe2deef6f05a" is_pre_election: true
I20260620 06:09:25.316728 26248 raft_consensus.cc:2454] T b729a999fd6b46d1af62153d893b6656 P 0d1347dff7a94072ad8cbe2deef6f05a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7c99aa7fa7834c89968adb1265fa3603 in term 0.
I20260620 06:09:25.316731 26110 raft_consensus.cc:2454] T b729a999fd6b46d1af62153d893b6656 P d7f871fdb6814adabd0538690e2a65a3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7c99aa7fa7834c89968adb1265fa3603 in term 0.
I20260620 06:09:25.317271 26316 leader_election.cc:302] T b729a999fd6b46d1af62153d893b6656 P 7c99aa7fa7834c89968adb1265fa3603 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 0d1347dff7a94072ad8cbe2deef6f05a, 7c99aa7fa7834c89968adb1265fa3603, d7f871fdb6814adabd0538690e2a65a3; no voters:
I20260620 06:09:25.317564 26444 raft_consensus.cc:2781] T b729a999fd6b46d1af62153d893b6656 P 7c99aa7fa7834c89968adb1265fa3603 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 06:09:25.317656 26444 raft_consensus.cc:482] T b729a999fd6b46d1af62153d893b6656 P 7c99aa7fa7834c89968adb1265fa3603 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 06:09:25.317700 26444 raft_consensus.cc:3037] T b729a999fd6b46d1af62153d893b6656 P 7c99aa7fa7834c89968adb1265fa3603 [term 0 FOLLOWER]: Advancing to term 1
I20260620 06:09:25.318799 26444 raft_consensus.cc:504] T b729a999fd6b46d1af62153d893b6656 P 7c99aa7fa7834c89968adb1265fa3603 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c99aa7fa7834c89968adb1265fa3603" member_type: VOTER last_known_addr { host: "127.11.102.3" port: 43859 } } peers { permanent_uuid: "0d1347dff7a94072ad8cbe2deef6f05a" member_type: VOTER last_known_addr { host: "127.11.102.2" port: 36731 } } peers { permanent_uuid: "d7f871fdb6814adabd0538690e2a65a3" member_type: VOTER last_known_addr { host: "127.11.102.1" port: 41679 } }
I20260620 06:09:25.319214 26444 leader_election.cc:288] T b729a999fd6b46d1af62153d893b6656 P 7c99aa7fa7834c89968adb1265fa3603 [CANDIDATE]: Term 1 election: Requested vote from peers 0d1347dff7a94072ad8cbe2deef6f05a (127.11.102.2:36731), d7f871fdb6814adabd0538690e2a65a3 (127.11.102.1:41679)
I20260620 06:09:25.319526 26248 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b729a999fd6b46d1af62153d893b6656" candidate_uuid: "7c99aa7fa7834c89968adb1265fa3603" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d1347dff7a94072ad8cbe2deef6f05a"
I20260620 06:09:25.319594 26110 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b729a999fd6b46d1af62153d893b6656" candidate_uuid: "7c99aa7fa7834c89968adb1265fa3603" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d7f871fdb6814adabd0538690e2a65a3"
I20260620 06:09:25.319702 26248 raft_consensus.cc:3037] T b729a999fd6b46d1af62153d893b6656 P 0d1347dff7a94072ad8cbe2deef6f05a [term 0 FOLLOWER]: Advancing to term 1
I20260620 06:09:25.319803 26110 raft_consensus.cc:3037] T b729a999fd6b46d1af62153d893b6656 P d7f871fdb6814adabd0538690e2a65a3 [term 0 FOLLOWER]: Advancing to term 1
I20260620 06:09:25.320796 26248 raft_consensus.cc:2454] T b729a999fd6b46d1af62153d893b6656 P 0d1347dff7a94072ad8cbe2deef6f05a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7c99aa7fa7834c89968adb1265fa3603 in term 1.
I20260620 06:09:25.320930 26110 raft_consensus.cc:2454] T b729a999fd6b46d1af62153d893b6656 P d7f871fdb6814adabd0538690e2a65a3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7c99aa7fa7834c89968adb1265fa3603 in term 1.
I20260620 06:09:25.321128 26316 leader_election.cc:302] T b729a999fd6b46d1af62153d893b6656 P 7c99aa7fa7834c89968adb1265fa3603 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d1347dff7a94072ad8cbe2deef6f05a, 7c99aa7fa7834c89968adb1265fa3603; no voters:
I20260620 06:09:25.321307 26444 raft_consensus.cc:2781] T b729a999fd6b46d1af62153d893b6656 P 7c99aa7fa7834c89968adb1265fa3603 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 06:09:25.321532 26444 raft_consensus.cc:686] T b729a999fd6b46d1af62153d893b6656 P 7c99aa7fa7834c89968adb1265fa3603 [term 1 LEADER]: Becoming Leader. State: Replica: 7c99aa7fa7834c89968adb1265fa3603, State: Running, Role: LEADER
I20260620 06:09:25.321863 26444 consensus_queue.cc:237] T b729a999fd6b46d1af62153d893b6656 P 7c99aa7fa7834c89968adb1265fa3603 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c99aa7fa7834c89968adb1265fa3603" member_type: VOTER last_known_addr { host: "127.11.102.3" port: 43859 } } peers { permanent_uuid: "0d1347dff7a94072ad8cbe2deef6f05a" member_type: VOTER last_known_addr { host: "127.11.102.2" port: 36731 } } peers { permanent_uuid: "d7f871fdb6814adabd0538690e2a65a3" member_type: VOTER last_known_addr { host: "127.11.102.1" port: 41679 } }
I20260620 06:09:25.325209 25966 catalog_manager.cc:5495] T b729a999fd6b46d1af62153d893b6656 P 7c99aa7fa7834c89968adb1265fa3603 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7c99aa7fa7834c89968adb1265fa3603 (127.11.102.3). New cstate: current_term: 1 leader_uuid: "7c99aa7fa7834c89968adb1265fa3603" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c99aa7fa7834c89968adb1265fa3603" member_type: VOTER last_known_addr { host: "127.11.102.3" port: 43859 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0d1347dff7a94072ad8cbe2deef6f05a" member_type: VOTER last_known_addr { host: "127.11.102.2" port: 36731 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d7f871fdb6814adabd0538690e2a65a3" member_type: VOTER last_known_addr { host: "127.11.102.1" port: 41679 } health_report { overall_health: UNKNOWN } } }
I20260620 06:09:25.338182 26435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:09:25.334565 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60002 (local address 127.11.102.3:43859)
0620 06:09:25.334732 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0620 06:09:25.334736 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 06:09:25.334750 (+ 14us) server_negotiation.cc:407] Connection header received
0620 06:09:25.334912 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:09:25.334916 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:09:25.334974 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:09:25.335088 (+ 114us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 06:09:25.335331 (+ 243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:09:25.335908 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:09:25.336745 (+ 837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:09:25.336963 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:09:25.337075 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:09:25.337378 (+ 303us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 06:09:25.337780 (+ 402us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 06:09:25.337922 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0620 06:09:25.337962 (+ 40us) server_negotiation.cc:299] Negotiation successful
0620 06:09:25.338016 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260620 06:09:25.344197 26110 raft_consensus.cc:1261] T b729a999fd6b46d1af62153d893b6656 P d7f871fdb6814adabd0538690e2a65a3 [term 1 FOLLOWER]: Refusing update from remote peer 7c99aa7fa7834c89968adb1265fa3603: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 06:09:25.344201 26248 raft_consensus.cc:1261] T b729a999fd6b46d1af62153d893b6656 P 0d1347dff7a94072ad8cbe2deef6f05a [term 1 FOLLOWER]: Refusing update from remote peer 7c99aa7fa7834c89968adb1265fa3603: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 06:09:25.344945 26444 consensus_queue.cc:1035] T b729a999fd6b46d1af62153d893b6656 P 7c99aa7fa7834c89968adb1265fa3603 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d1347dff7a94072ad8cbe2deef6f05a" member_type: VOTER last_known_addr { host: "127.11.102.2" port: 36731 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 06:09:25.345216 26446 consensus_queue.cc:1035] T b729a999fd6b46d1af62153d893b6656 P 7c99aa7fa7834c89968adb1265fa3603 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7f871fdb6814adabd0538690e2a65a3" member_type: VOTER last_known_addr { host: "127.11.102.1" port: 41679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 06:09:25.352103 26449 mvcc.cc:205] Tried to move back new op lower bound from 7298808894842277888 to 7298808894761271296. Current Snapshot: MvccSnapshot[applied={T|T < 7298808894842277888}]
I20260620 06:09:25.353269 26450 mvcc.cc:205] Tried to move back new op lower bound from 7298808894842277888 to 7298808894761271296. Current Snapshot: MvccSnapshot[applied={T|T < 7298808894842277888}]
I20260620 06:09:25.353793 26451 mvcc.cc:205] Tried to move back new op lower bound from 7298808894842277888 to 7298808894761271296. Current Snapshot: MvccSnapshot[applied={T|T < 7298808894842277888}]
I20260620 06:09:25.360585 25966 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:48664:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 06:09:25.360715 25966 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:48664:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 06:09:25.362910 25966 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6743bd85c057436695ea713d0e6cd57b: Sending DeleteTablet for 3 replicas of tablet b729a999fd6b46d1af62153d893b6656
I20260620 06:09:25.363628 26360 tablet_service.cc:1514] Processing DeleteTablet for tablet b729a999fd6b46d1af62153d893b6656 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 06:09:25 UTC) from {username='kudu', principal='kudu/127.11.102.62@KRBTEST.COM'} at 127.0.0.1:59988
I20260620 06:09:25.363720 26228 tablet_service.cc:1514] Processing DeleteTablet for tablet b729a999fd6b46d1af62153d893b6656 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 06:09:25 UTC) from {username='kudu', principal='kudu/127.11.102.62@KRBTEST.COM'} at 127.0.0.1:40030
I20260620 06:09:25.363826 26090 tablet_service.cc:1514] Processing DeleteTablet for tablet b729a999fd6b46d1af62153d893b6656 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 06:09:25 UTC) from {username='kudu', principal='kudu/127.11.102.62@KRBTEST.COM'} at 127.0.0.1:50764
I20260620 06:09:25.364167 26460 tablet_replica.cc:330] T b729a999fd6b46d1af62153d893b6656 P 0d1347dff7a94072ad8cbe2deef6f05a: stopping tablet replica
I20260620 06:09:25.364312 11672 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9HMp43/build/debug/bin/kudu with pid 26006
I20260620 06:09:25.364482 26460 raft_consensus.cc:2229] T b729a999fd6b46d1af62153d893b6656 P 0d1347dff7a94072ad8cbe2deef6f05a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 06:09:25.366307 26460 raft_consensus.cc:2258] T b729a999fd6b46d1af62153d893b6656 P 0d1347dff7a94072ad8cbe2deef6f05a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 06:09:25.366710 26458 tablet_replica.cc:330] T b729a999fd6b46d1af62153d893b6656 P 7c99aa7fa7834c89968adb1265fa3603: stopping tablet replica
I20260620 06:09:25.367002 26458 raft_consensus.cc:2229] T b729a999fd6b46d1af62153d893b6656 P 7c99aa7fa7834c89968adb1265fa3603 [term 1 LEADER]: Raft consensus shutting down.
I20260620 06:09:25.367394 26458 raft_consensus.cc:2258] T b729a999fd6b46d1af62153d893b6656 P 7c99aa7fa7834c89968adb1265fa3603 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 06:09:25.367851 26460 ts_tablet_manager.cc:1905] T b729a999fd6b46d1af62153d893b6656 P 0d1347dff7a94072ad8cbe2deef6f05a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 06:09:25.370004 26460 ts_tablet_manager.cc:1918] T b729a999fd6b46d1af62153d893b6656 P 0d1347dff7a94072ad8cbe2deef6f05a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 06:09:25.370112 26460 log.cc:1196] T b729a999fd6b46d1af62153d893b6656 P 0d1347dff7a94072ad8cbe2deef6f05a: Deleting WAL directory at /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935603120653-11672-0/minicluster-data/ts-1/wal/wals/b729a999fd6b46d1af62153d893b6656
I20260620 06:09:25.370419 26460 ts_tablet_manager.cc:1939] T b729a999fd6b46d1af62153d893b6656 P 0d1347dff7a94072ad8cbe2deef6f05a: Deleting consensus metadata
I20260620 06:09:25.371543 25955 catalog_manager.cc:4841] TS 0d1347dff7a94072ad8cbe2deef6f05a (127.11.102.2:36731): tablet b729a999fd6b46d1af62153d893b6656 (table test-table [id=bc583ad1632a474d89a40ad17502dc09]) successfully deleted
I20260620 06:09:25.372664 26458 ts_tablet_manager.cc:1905] T b729a999fd6b46d1af62153d893b6656 P 7c99aa7fa7834c89968adb1265fa3603: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260620 06:09:25.373335 25952 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.11.102.1:41679 (error 108)
I20260620 06:09:25.373613 11672 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9HMp43/build/debug/bin/kudu with pid 26161
I20260620 06:09:25.376267 26436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:09:25.375934 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.102.1:41679 (local address 127.0.0.1:50766)
0620 06:09:25.376094 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0620 06:09:25.376170 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.102.1:41679: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
W20260620 06:09:25.376510 25952 catalog_manager.cc:4584] TS d7f871fdb6814adabd0538690e2a65a3 (127.11.102.1:41679): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet b729a999fd6b46d1af62153d893b6656: Network error: Client connection negotiation failed: client connection to 127.11.102.1:41679: connect: Connection refused (error 111)
I20260620 06:09:25.377393 26458 ts_tablet_manager.cc:1918] T b729a999fd6b46d1af62153d893b6656 P 7c99aa7fa7834c89968adb1265fa3603: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 06:09:25.377506 26458 log.cc:1196] T b729a999fd6b46d1af62153d893b6656 P 7c99aa7fa7834c89968adb1265fa3603: Deleting WAL directory at /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935603120653-11672-0/minicluster-data/ts-2/wal/wals/b729a999fd6b46d1af62153d893b6656
I20260620 06:09:25.377802 26458 ts_tablet_manager.cc:1939] T b729a999fd6b46d1af62153d893b6656 P 7c99aa7fa7834c89968adb1265fa3603: Deleting consensus metadata
I20260620 06:09:25.378840 25954 catalog_manager.cc:4841] TS 7c99aa7fa7834c89968adb1265fa3603 (127.11.102.3:43859): tablet b729a999fd6b46d1af62153d893b6656 (table test-table [id=bc583ad1632a474d89a40ad17502dc09]) successfully deleted
I20260620 06:09:25.381570 11672 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9HMp43/build/debug/bin/kudu with pid 26297
I20260620 06:09:25.386476 11672 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9HMp43/build/debug/bin/kudu with pid 25935
I20260620 06:09:25.391955 11672 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9HMp43/build/debug/bin/kudu
/tmp/dist-test-task9HMp43/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935603120653-11672-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935603120653-11672-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935603120653-11672-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935603120653-11672-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.102.62:41475
--webserver_interface=127.11.102.62
--webserver_port=43041
--keytab_file=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935603120653-11672-0/krb5kdc/kudu_127.11.102.62.keytab
--principal=kudu/127.11.102.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.102.20:37051
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.102.62:41475
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935603120653-11672-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935603120653-11672-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935603120653-11672-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 06:09:25.504302 26462 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:09:25.504606 26462 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:09:25.504693 26462 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:09:25.508679 26462 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 06:09:25.508780 26462 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:09:25.508821 26462 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:09:25.508884 26462 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 06:09:25.508920 26462 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 06:09:25.514083 26462 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.102.20:37051
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935603120653-11672-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935603120653-11672-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.102.62:41475
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935603120653-11672-0/krb5kdc/kudu_127.11.102.62.keytab
--principal=kudu/127.11.102.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.102.62:41475
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935603120653-11672-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.102.62
--webserver_port=43041
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26462
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935603120653-11672-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 05:28:48 UTC on ccycloud.quasar-fixhms.root.comops.site
build id 4457020
I20260620 06:09:25.515139 26462 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:09:25.516155 26462 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 06:09:25.524622 26467 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:09:25.524619 26470 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:09:25.524806 26468 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:09:25.524842 26462 server_base.cc:1034] running on GCE node
I20260620 06:09:25.525552 26462 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:09:25.526605 26462 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:09:25.527801 26462 hybrid_clock.cc:648] HybridClock initialized: now 1781935765527771 us; error 49 us; skew 500 ppm
Jun 20 06:09:25 dist-test-slave-9hm5 krb5kdc[25389](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935765, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.102.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 06:09:25.535276 26462 init.cc:377] Logged in from keytab as kudu/127.11.102.62@KRBTEST.COM (short username kudu)
I20260620 06:09:25.536259 26462 webserver.cc:460] Webserver started at http://127.11.102.62:43041/ using document root <none> and password file <none>
I20260620 06:09:25.536777 26462 fs_manager.cc:359] Metadata directory not provided
I20260620 06:09:25.536837 26462 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:09:25.540542 26462 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260620 06:09:25.542559 26477 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:09:25.543825 26462 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 06:09:25.543927 26462 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935603120653-11672-0/minicluster-data/master-0/data,/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935603120653-11672-0/minicluster-data/master-0/wal
uuid: "6743bd85c057436695ea713d0e6cd57b"
format_stamp: "Formatted at 2026-06-20 06:09:23 on dist-test-slave-9hm5"
server_key: "bc428878fb4faafd9bd236886d35abf7"
server_key_iv: "aa844420908e3443787e4f2fb6edd13e"
server_key_version: "encryptionkey@0"
I20260620 06:09:25.544433 26462 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935603120653-11672-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935603120653-11672-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935603120653-11672-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:09:25.562393 26462 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:09:25.565481 26462 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:09:25.565641 26462 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:09:25.572522 26462 rpc_server.cc:307] RPC server started. Bound to: 127.11.102.62:41475
I20260620 06:09:25.572801 26529 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.102.62:41475 every 8 connection(s)
I20260620 06:09:25.573606 26462 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935603120653-11672-0/minicluster-data/master-0/data/info.pb
I20260620 06:09:25.577771 26530 sys_catalog.cc:263] Verifying existing consensus state
I20260620 06:09:25.578699 11672 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9HMp43/build/debug/bin/kudu as pid 26462
I20260620 06:09:25.579157 11672 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9HMp43/build/debug/bin/kudu
/tmp/dist-test-task9HMp43/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935603120653-11672-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935603120653-11672-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935603120653-11672-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935603120653-11672-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.102.1:41679
--local_ip_for_outbound_sockets=127.11.102.1
--tserver_master_addrs=127.11.102.62:41475
--webserver_port=42581
--webserver_interface=127.11.102.1
--keytab_file=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935603120653-11672-0/krb5kdc/kudu_127.11.102.1.keytab
--principal=kudu/127.11.102.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.102.20:37051
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935603120653-11672-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935603120653-11672-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935603120653-11672-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 06:09:25.579514 26530 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6743bd85c057436695ea713d0e6cd57b: Bootstrap starting.
I20260620 06:09:25.596511 26530 log.cc:824] T 00000000000000000000000000000000 P 6743bd85c057436695ea713d0e6cd57b: Log is configured to *not* fsync() on all Append() calls
I20260620 06:09:25.607510 26530 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6743bd85c057436695ea713d0e6cd57b: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 06:09:25.607915 26530 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6743bd85c057436695ea713d0e6cd57b: Bootstrap complete.
I20260620 06:09:25.611949 26530 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6743bd85c057436695ea713d0e6cd57b [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6743bd85c057436695ea713d0e6cd57b" member_type: VOTER last_known_addr { host: "127.11.102.62" port: 41475 } }
I20260620 06:09:25.612383 26530 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6743bd85c057436695ea713d0e6cd57b [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6743bd85c057436695ea713d0e6cd57b, State: Initialized, Role: FOLLOWER
I20260620 06:09:25.612924 26530 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6743bd85c057436695ea713d0e6cd57b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6743bd85c057436695ea713d0e6cd57b" member_type: VOTER last_known_addr { host: "127.11.102.62" port: 41475 } }
I20260620 06:09:25.613073 26530 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6743bd85c057436695ea713d0e6cd57b [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 06:09:25.613143 26530 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6743bd85c057436695ea713d0e6cd57b [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 06:09:25.613234 26530 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6743bd85c057436695ea713d0e6cd57b [term 2 FOLLOWER]: Advancing to term 3
I20260620 06:09:25.615049 26530 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6743bd85c057436695ea713d0e6cd57b [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6743bd85c057436695ea713d0e6cd57b" member_type: VOTER last_known_addr { host: "127.11.102.62" port: 41475 } }
I20260620 06:09:25.615381 26530 leader_election.cc:302] T 00000000000000000000000000000000 P 6743bd85c057436695ea713d0e6cd57b [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6743bd85c057436695ea713d0e6cd57b; no voters:
I20260620 06:09:25.615754 26530 leader_election.cc:288] T 00000000000000000000000000000000 P 6743bd85c057436695ea713d0e6cd57b [CANDIDATE]: Term 3 election: Requested vote from peers
I20260620 06:09:25.615885 26534 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6743bd85c057436695ea713d0e6cd57b [term 3 FOLLOWER]: Leader election won for term 3
I20260620 06:09:25.616148 26534 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6743bd85c057436695ea713d0e6cd57b [term 3 LEADER]: Becoming Leader. State: Replica: 6743bd85c057436695ea713d0e6cd57b, State: Running, Role: LEADER
I20260620 06:09:25.616452 26534 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6743bd85c057436695ea713d0e6cd57b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6743bd85c057436695ea713d0e6cd57b" member_type: VOTER last_known_addr { host: "127.11.102.62" port: 41475 } }
I20260620 06:09:25.616952 26530 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6743bd85c057436695ea713d0e6cd57b [sys.catalog]: configured and running, proceeding with master startup.
I20260620 06:09:25.617797 26535 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6743bd85c057436695ea713d0e6cd57b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "6743bd85c057436695ea713d0e6cd57b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6743bd85c057436695ea713d0e6cd57b" member_type: VOTER last_known_addr { host: "127.11.102.62" port: 41475 } } }
I20260620 06:09:25.617926 26535 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6743bd85c057436695ea713d0e6cd57b [sys.catalog]: This master's current role is: LEADER
I20260620 06:09:25.618557 26536 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6743bd85c057436695ea713d0e6cd57b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6743bd85c057436695ea713d0e6cd57b. Latest consensus state: current_term: 3 leader_uuid: "6743bd85c057436695ea713d0e6cd57b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6743bd85c057436695ea713d0e6cd57b" member_type: VOTER last_known_addr { host: "127.11.102.62" port: 41475 } } }
I20260620 06:09:25.618729 26536 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6743bd85c057436695ea713d0e6cd57b [sys.catalog]: This master's current role is: LEADER
I20260620 06:09:25.618924 26543 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 06:09:25.624318 26543 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 06:09:25.624917 26543 catalog_manager.cc:1259] Loaded cluster ID: c8554031cfe44467b49feb636b18c097
I20260620 06:09:25.624977 26543 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260620 06:09:25.625918 26543 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f22673f89ad google::LogMessage::Fail() at ??:0
@ 0x7f22673fb88a google::LogMessage::SendToLog() at ??:0
@ 0x7f22673f84de google::LogMessage::Flush() at ??:0
@ 0x7f22673f9a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f2271d37fdd _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_ at ??:0
@ 0x7f2271d38d2f kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f2271d35579 _ZZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvENKUlvE_clEv at ??:0
@ 0x7f2271d79640 _ZNSt17_Function_handlerIFvvEZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f226964a3d7 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f226964ac83 _ZZN4kudu10ThreadPool12CreateThreadEvENKUlvE_clEv at ??:0
@ 0x7f226964c3f2 _ZNSt17_Function_handlerIFvvEZN4kudu10ThreadPool12CreateThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f226963b512 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f226ac7f6db start_thread at ??:0
@ 0x7f2265ef661f clone at ??:0
W20260620 06:09:25.709637 26532 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:09:25.709894 26532 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:09:25.709976 26532 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:09:25.714064 26532 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:09:25.714148 26532 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:09:25.714267 26532 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.102.1
I20260620 06:09:25.719481 26532 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.102.20:37051
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935603120653-11672-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935603120653-11672-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935603120653-11672-0/krb5kdc/kudu_127.11.102.1.keytab
--principal=kudu/127.11.102.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.102.1:41679
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935603120653-11672-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.11.102.1
--webserver_port=42581
--tserver_master_addrs=127.11.102.62:41475
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26532
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.102.1
--log_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935603120653-11672-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 05:28:48 UTC on ccycloud.quasar-fixhms.root.comops.site
build id 4457020
I20260620 06:09:25.720556 26532 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:09:25.721510 26532 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 06:09:25.730767 26571 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:09:25.730777 26574 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:09:25.730777 26572 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:09:25.731070 26532 server_base.cc:1034] running on GCE node
I20260620 06:09:25.731659 26532 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:09:25.732363 26532 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:09:25.733572 26532 hybrid_clock.cc:648] HybridClock initialized: now 1781935765733549 us; error 31 us; skew 500 ppm
Jun 20 06:09:25 dist-test-slave-9hm5 krb5kdc[25389](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935765, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.102.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 06:09:25.736713 26532 init.cc:377] Logged in from keytab as kudu/127.11.102.1@KRBTEST.COM (short username kudu)
I20260620 06:09:25.737501 26532 webserver.cc:460] Webserver started at http://127.11.102.1:42581/ using document root <none> and password file <none>
I20260620 06:09:25.738035 26532 fs_manager.cc:359] Metadata directory not provided
I20260620 06:09:25.738129 26532 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:09:25.741945 26532 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.000s sys 0.003s
I20260620 06:09:25.744140 26581 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:09:25.745512 26532 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260620 06:09:25.745671 26532 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935603120653-11672-0/minicluster-data/ts-0/data,/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935603120653-11672-0/minicluster-data/ts-0/wal
uuid: "d7f871fdb6814adabd0538690e2a65a3"
format_stamp: "Formatted at 2026-06-20 06:09:23 on dist-test-slave-9hm5"
server_key: "110aac0377228e58ef7c06a5891315b6"
server_key_iv: "389c4dcb9928faf9e8cd6baa4e0b02f5"
server_key_version: "encryptionkey@0"
I20260620 06:09:25.746217 26532 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935603120653-11672-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935603120653-11672-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935603120653-11672-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:09:25.773442 26532 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:09:25.776978 26532 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:09:25.777210 26532 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:09:25.777828 26532 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 06:09:25.778992 26588 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260620 06:09:25.784641 26532 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260620 06:09:25.784740 26532 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.006s user 0.000s sys 0.000s
I20260620 06:09:25.784821 26532 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260620 06:09:25.786458 26532 ts_tablet_manager.cc:610] Registered 1 tablets
I20260620 06:09:25.786532 26532 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260620 06:09:25.786854 26588 tablet_bootstrap.cc:492] T b729a999fd6b46d1af62153d893b6656 P d7f871fdb6814adabd0538690e2a65a3: Bootstrap starting.
I20260620 06:09:25.797180 26532 rpc_server.cc:307] RPC server started. Bound to: 127.11.102.1:41679
I20260620 06:09:25.797425 26695 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.102.1:41679 every 8 connection(s)
I20260620 06:09:25.798199 26532 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935603120653-11672-0/minicluster-data/ts-0/data/info.pb
I20260620 06:09:25.805998 11672 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9HMp43/build/debug/bin/kudu as pid 26532
I20260620 06:09:25.806427 11672 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9HMp43/build/debug/bin/kudu
/tmp/dist-test-task9HMp43/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935603120653-11672-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935603120653-11672-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935603120653-11672-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935603120653-11672-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.102.2:36731
--local_ip_for_outbound_sockets=127.11.102.2
--tserver_master_addrs=127.11.102.62:41475
--webserver_port=42277
--webserver_interface=127.11.102.2
--keytab_file=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935603120653-11672-0/krb5kdc/kudu_127.11.102.2.keytab
--principal=kudu/127.11.102.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.102.20:37051
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935603120653-11672-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935603120653-11672-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935603120653-11672-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 06:09:25.808597 26588 log.cc:824] T b729a999fd6b46d1af62153d893b6656 P d7f871fdb6814adabd0538690e2a65a3: Log is configured to *not* fsync() on all Append() calls
I20260620 06:09:25.817528 26588 tablet_bootstrap.cc:492] T b729a999fd6b46d1af62153d893b6656 P d7f871fdb6814adabd0538690e2a65a3: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 06:09:25.817982 26588 tablet_bootstrap.cc:492] T b729a999fd6b46d1af62153d893b6656 P d7f871fdb6814adabd0538690e2a65a3: Bootstrap complete.
I20260620 06:09:25.819235 26588 ts_tablet_manager.cc:1397] T b729a999fd6b46d1af62153d893b6656 P d7f871fdb6814adabd0538690e2a65a3: Time spent bootstrapping tablet: real 0.033s user 0.025s sys 0.004s
I20260620 06:09:25.822722 26588 raft_consensus.cc:348] T b729a999fd6b46d1af62153d893b6656 P d7f871fdb6814adabd0538690e2a65a3 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c99aa7fa7834c89968adb1265fa3603" member_type: VOTER last_known_addr { host: "127.11.102.3" port: 43859 } } peers { permanent_uuid: "0d1347dff7a94072ad8cbe2deef6f05a" member_type: VOTER last_known_addr { host: "127.11.102.2" port: 36731 } } peers { permanent_uuid: "d7f871fdb6814adabd0538690e2a65a3" member_type: VOTER last_known_addr { host: "127.11.102.1" port: 41679 } }
I20260620 06:09:25.823290 26588 raft_consensus.cc:729] T b729a999fd6b46d1af62153d893b6656 P d7f871fdb6814adabd0538690e2a65a3 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: d7f871fdb6814adabd0538690e2a65a3, State: Initialized, Role: FOLLOWER
I20260620 06:09:25.823855 26588 consensus_queue.cc:260] T b729a999fd6b46d1af62153d893b6656 P d7f871fdb6814adabd0538690e2a65a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c99aa7fa7834c89968adb1265fa3603" member_type: VOTER last_known_addr { host: "127.11.102.3" port: 43859 } } peers { permanent_uuid: "0d1347dff7a94072ad8cbe2deef6f05a" member_type: VOTER last_known_addr { host: "127.11.102.2" port: 36731 } } peers { permanent_uuid: "d7f871fdb6814adabd0538690e2a65a3" member_type: VOTER last_known_addr { host: "127.11.102.1" port: 41679 } }
I20260620 06:09:25.824954 26588 ts_tablet_manager.cc:1428] T b729a999fd6b46d1af62153d893b6656 P d7f871fdb6814adabd0538690e2a65a3: Time spent starting tablet: real 0.006s user 0.008s sys 0.000s
W20260620 06:09:25.871950 26698 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 06:09:25.800205 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.102.62:41475 (local address 127.11.102.1:49461)
0620 06:09:25.800682 (+ 477us) negotiation.cc:107] Waiting for socket to connect
0620 06:09:25.800733 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0620 06:09:25.801523 (+ 790us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:09:25.871005 (+ 69482us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.102.62:41475: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":267,"thread_start_us":136,"threads_started":1}
W20260620 06:09:25.872594 26696 heartbeater.cc:644] Failed to heartbeat to 127.11.102.62:41475 (0 consecutive failures): Network error: Failed to ping master at 127.11.102.62:41475: Client connection negotiation failed: client connection to 127.11.102.62:41475: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260620 06:09:25.873368 26698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:09:25.873074 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.102.62:41475 (local address 127.11.102.1:46779)
0620 06:09:25.873230 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0620 06:09:25.873305 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.102.62:41475: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
I20260620 06:09:25.874038 26698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:09:25.873832 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.102.62:41475 (local address 127.11.102.1:46133)
0620 06:09:25.873940 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0620 06:09:25.873986 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.102.62:41475: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260620 06:09:25.874235 26696 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260620 06:09:25.916939 26699 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:09:25.917184 26699 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:09:25.917227 26699 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:09:25.921495 26699 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:09:25.921556 26699 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:09:25.921658 26699 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.102.2
I20260620 06:09:25.927016 26699 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.102.20:37051
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935603120653-11672-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935603120653-11672-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935603120653-11672-0/krb5kdc/kudu_127.11.102.2.keytab
--principal=kudu/127.11.102.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.102.2:36731
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935603120653-11672-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.11.102.2
--webserver_port=42277
--tserver_master_addrs=127.11.102.62:41475
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26699
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.102.2
--log_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935603120653-11672-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 05:28:48 UTC on ccycloud.quasar-fixhms.root.comops.site
build id 4457020
I20260620 06:09:25.928135 26699 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:09:25.929102 26699 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 06:09:25.938585 26699 server_base.cc:1034] running on GCE node
W20260620 06:09:25.938578 26707 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:09:25.938575 26709 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:09:25.938768 26706 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:09:25.939232 26699 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:09:25.939914 26699 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:09:25.941073 26699 hybrid_clock.cc:648] HybridClock initialized: now 1781935765941059 us; error 26 us; skew 500 ppm
Jun 20 06:09:25 dist-test-slave-9hm5 krb5kdc[25389](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935765, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.102.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 06:09:25.943863 26699 init.cc:377] Logged in from keytab as kudu/127.11.102.2@KRBTEST.COM (short username kudu)
I20260620 06:09:25.944654 26699 webserver.cc:460] Webserver started at http://127.11.102.2:42277/ using document root <none> and password file <none>
I20260620 06:09:25.945194 26699 fs_manager.cc:359] Metadata directory not provided
I20260620 06:09:25.945291 26699 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:09:25.948872 26699 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.001s
I20260620 06:09:25.950999 26716 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:09:25.952168 26699 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260620 06:09:25.952298 26699 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935603120653-11672-0/minicluster-data/ts-1/data,/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935603120653-11672-0/minicluster-data/ts-1/wal
uuid: "0d1347dff7a94072ad8cbe2deef6f05a"
format_stamp: "Formatted at 2026-06-20 06:09:23 on dist-test-slave-9hm5"
server_key: "6dd3b8101a03a814b1a68fe753780af8"
server_key_iv: "9bcd1732e4c77ea13bb2b9fadd277fca"
server_key_version: "encryptionkey@0"
I20260620 06:09:25.952819 26699 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935603120653-11672-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935603120653-11672-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935603120653-11672-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:09:25.970825 26699 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:09:25.973759 26699 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:09:25.973960 26699 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:09:25.974524 26699 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 06:09:25.975314 26699 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 06:09:25.975389 26699 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 06:09:25.975456 26699 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 06:09:25.975488 26699 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 06:09:25.984760 26699 rpc_server.cc:307] RPC server started. Bound to: 127.11.102.2:36731
I20260620 06:09:25.984778 26829 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.102.2:36731 every 8 connection(s)
I20260620 06:09:25.985689 26699 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935603120653-11672-0/minicluster-data/ts-1/data/info.pb
I20260620 06:09:25.989220 26832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:09:25.987592 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.102.62:41475 (local address 127.11.102.2:48581)
0620 06:09:25.988123 (+ 531us) negotiation.cc:107] Waiting for socket to connect
0620 06:09:25.988526 (+ 403us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.102.62:41475: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":279,"thread_start_us":109,"threads_started":1}
W20260620 06:09:25.989676 26830 heartbeater.cc:644] Failed to heartbeat to 127.11.102.62:41475 (0 consecutive failures): Network error: Failed to ping master at 127.11.102.62:41475: Client connection negotiation failed: client connection to 127.11.102.62:41475: connect: Connection refused (error 111)
I20260620 06:09:25.990240 26832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:09:25.990036 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.102.62:41475 (local address 127.11.102.2:43401)
0620 06:09:25.990144 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0620 06:09:25.990188 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.102.62:41475: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260620 06:09:25.990792 26832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:09:25.990615 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.102.62:41475 (local address 127.11.102.2:40577)
0620 06:09:25.990705 (+ 90us) negotiation.cc:107] Waiting for socket to connect
0620 06:09:25.990744 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.102.62:41475: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260620 06:09:25.990967 26830 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260620 06:09:25.993028 11672 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9HMp43/build/debug/bin/kudu as pid 26699
I20260620 06:09:25.993429 11672 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9HMp43/build/debug/bin/kudu
/tmp/dist-test-task9HMp43/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935603120653-11672-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935603120653-11672-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935603120653-11672-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935603120653-11672-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.102.3:43859
--local_ip_for_outbound_sockets=127.11.102.3
--tserver_master_addrs=127.11.102.62:41475
--webserver_port=33315
--webserver_interface=127.11.102.3
--keytab_file=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935603120653-11672-0/krb5kdc/kudu_127.11.102.3.keytab
--principal=kudu/127.11.102.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.102.20:37051
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935603120653-11672-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935603120653-11672-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935603120653-11672-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 06:09:26.098726 26833 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:09:26.098942 26833 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:09:26.098982 26833 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:09:26.102830 26833 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:09:26.102880 26833 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:09:26.102988 26833 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.102.3
I20260620 06:09:26.107789 26833 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.102.20:37051
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935603120653-11672-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935603120653-11672-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935603120653-11672-0/krb5kdc/kudu_127.11.102.3.keytab
--principal=kudu/127.11.102.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.102.3:43859
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935603120653-11672-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.11.102.3
--webserver_port=33315
--tserver_master_addrs=127.11.102.62:41475
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26833
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.102.3
--log_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935603120653-11672-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 05:28:48 UTC on ccycloud.quasar-fixhms.root.comops.site
build id 4457020
I20260620 06:09:26.108680 26833 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:09:26.109666 26833 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 06:09:26.119026 26839 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:09:26.119062 26838 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:09:26.119105 26841 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:09:26.119107 26833 server_base.cc:1034] running on GCE node
I20260620 06:09:26.119774 26833 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:09:26.120419 26833 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:09:26.121613 26833 hybrid_clock.cc:648] HybridClock initialized: now 1781935766121601 us; error 28 us; skew 500 ppm
Jun 20 06:09:26 dist-test-slave-9hm5 krb5kdc[25389](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935766, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.102.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 06:09:26.124891 26833 init.cc:377] Logged in from keytab as kudu/127.11.102.3@KRBTEST.COM (short username kudu)
I20260620 06:09:26.125784 26833 webserver.cc:460] Webserver started at http://127.11.102.3:33315/ using document root <none> and password file <none>
I20260620 06:09:26.126341 26833 fs_manager.cc:359] Metadata directory not provided
I20260620 06:09:26.126437 26833 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:09:26.130127 26833 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.001s sys 0.000s
I20260620 06:09:26.132221 26848 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:09:26.133414 26833 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 06:09:26.133560 26833 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935603120653-11672-0/minicluster-data/ts-2/data,/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935603120653-11672-0/minicluster-data/ts-2/wal
uuid: "7c99aa7fa7834c89968adb1265fa3603"
format_stamp: "Formatted at 2026-06-20 06:09:24 on dist-test-slave-9hm5"
server_key: "b3b0ed89c1cc4c95c4ba805a87b5e8c5"
server_key_iv: "57f8a6a192208426f9de3dc1b98d0013"
server_key_version: "encryptionkey@0"
I20260620 06:09:26.134090 26833 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935603120653-11672-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935603120653-11672-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935603120653-11672-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:09:26.200134 26833 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:09:26.203308 26833 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:09:26.203531 26833 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:09:26.204169 26833 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 06:09:26.205024 26833 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 06:09:26.205101 26833 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 06:09:26.205169 26833 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 06:09:26.205227 26833 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 06:09:26.214254 26833 rpc_server.cc:307] RPC server started. Bound to: 127.11.102.3:43859
I20260620 06:09:26.214323 26961 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.102.3:43859 every 8 connection(s)
I20260620 06:09:26.215296 26833 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935603120653-11672-0/minicluster-data/ts-2/data/info.pb
I20260620 06:09:26.219168 26964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:09:26.217597 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.102.62:41475 (local address 127.11.102.3:57169)
0620 06:09:26.218028 (+ 431us) negotiation.cc:107] Waiting for socket to connect
0620 06:09:26.218455 (+ 427us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.102.62:41475: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":254,"thread_start_us":106,"threads_started":1}
W20260620 06:09:26.219744 26962 heartbeater.cc:644] Failed to heartbeat to 127.11.102.62:41475 (0 consecutive failures): Network error: Failed to ping master at 127.11.102.62:41475: Client connection negotiation failed: client connection to 127.11.102.62:41475: connect: Connection refused (error 111)
I20260620 06:09:26.220105 11672 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9HMp43/build/debug/bin/kudu as pid 26833
I20260620 06:09:26.220317 26964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:09:26.220094 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.102.62:41475 (local address 127.11.102.3:49337)
0620 06:09:26.220202 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0620 06:09:26.220251 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.102.62:41475: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20260620 06:09:26.220945 26964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:09:26.220745 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.102.62:41475 (local address 127.11.102.3:51525)
0620 06:09:26.220853 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0620 06:09:26.220895 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.102.62:41475: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260620 06:09:26.221184 26962 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260620 06:09:26.875355 26966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:09:26.874894 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.102.62:41475 (local address 127.11.102.1:51975)
0620 06:09:26.875185 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0620 06:09:26.875245 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.102.62:41475: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":218,"thread_start_us":90,"threads_started":1}
I20260620 06:09:26.991928 26967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:09:26.991483 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.102.62:41475 (local address 127.11.102.2:44223)
0620 06:09:26.991783 (+ 300us) negotiation.cc:107] Waiting for socket to connect
0620 06:09:26.991833 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.102.62:41475: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":235,"thread_start_us":97,"threads_started":1}
I20260620 06:09:27.222179 26968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:09:27.221771 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.102.62:41475 (local address 127.11.102.3:49989)
0620 06:09:27.222030 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0620 06:09:27.222085 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.102.62:41475: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":91,"threads_started":1}
I20260620 06:09:27.368427 26969 raft_consensus.cc:482] T b729a999fd6b46d1af62153d893b6656 P d7f871fdb6814adabd0538690e2a65a3 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 06:09:27.368671 26969 raft_consensus.cc:504] T b729a999fd6b46d1af62153d893b6656 P d7f871fdb6814adabd0538690e2a65a3 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c99aa7fa7834c89968adb1265fa3603" member_type: VOTER last_known_addr { host: "127.11.102.3" port: 43859 } } peers { permanent_uuid: "0d1347dff7a94072ad8cbe2deef6f05a" member_type: VOTER last_known_addr { host: "127.11.102.2" port: 36731 } } peers { permanent_uuid: "d7f871fdb6814adabd0538690e2a65a3" member_type: VOTER last_known_addr { host: "127.11.102.1" port: 41679 } }
I20260620 06:09:27.369823 26969 leader_election.cc:288] T b729a999fd6b46d1af62153d893b6656 P d7f871fdb6814adabd0538690e2a65a3 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 7c99aa7fa7834c89968adb1265fa3603 (127.11.102.3:43859), 0d1347dff7a94072ad8cbe2deef6f05a (127.11.102.2:36731)
Jun 20 06:09:27 dist-test-slave-9hm5 krb5kdc[25389](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935765, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.102.1@KRBTEST.COM for kudu/127.11.102.3@KRBTEST.COM
Jun 20 06:09:27 dist-test-slave-9hm5 krb5kdc[25389](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935765, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.102.1@KRBTEST.COM for kudu/127.11.102.2@KRBTEST.COM
I20260620 06:09:27.384048 26971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:09:27.370255 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.102.1:44665 (local address 127.11.102.3:43859)
0620 06:09:27.370862 (+ 607us) server_negotiation.cc:206] Beginning negotiation
0620 06:09:27.370882 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0620 06:09:27.370916 (+ 34us) server_negotiation.cc:407] Connection header received
0620 06:09:27.371556 (+ 640us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:09:27.371562 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:09:27.371942 (+ 380us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:09:27.372252 (+ 310us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:09:27.373477 (+ 1225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:09:27.374470 (+ 993us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:09:27.375715 (+ 1245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:09:27.376184 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:09:27.376381 (+ 197us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:09:27.379230 (+ 2849us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:09:27.379235 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:09:27.379246 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:09:27.379270 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:09:27.381198 (+ 1928us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:09:27.381728 (+ 530us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:09:27.381760 (+ 32us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:09:27.381768 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:09:27.381920 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:09:27.382382 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:09:27.382385 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:09:27.382387 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:09:27.382820 (+ 433us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:09:27.382920 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0620 06:09:27.383548 (+ 628us) server_negotiation.cc:299] Negotiation successful
0620 06:09:27.383849 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":508,"thread_start_us":87,"threads_started":1}
I20260620 06:09:27.384060 26966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:09:27.370073 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.102.2:36731 (local address 127.11.102.1:42191)
0620 06:09:27.370240 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0620 06:09:27.370255 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 06:09:27.370360 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:09:27.372272 (+ 1912us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:09:27.372279 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:09:27.372846 (+ 567us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:09:27.373296 (+ 450us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:09:27.373308 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:09:27.374594 (+ 1286us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:09:27.374601 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:09:27.375544 (+ 943us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:09:27.375555 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:09:27.376301 (+ 746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:09:27.376305 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:09:27.376558 (+ 253us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:09:27.377260 (+ 702us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:09:27.377277 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:09:27.379384 (+ 2107us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:09:27.381786 (+ 2402us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:09:27.381789 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:09:27.381791 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:09:27.382013 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:09:27.382382 (+ 369us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:09:27.382385 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:09:27.382387 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:09:27.382477 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:09:27.383147 (+ 670us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:09:27.383150 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:09:27.383232 (+ 82us) client_negotiation.cc:769] Sending connection context
0620 06:09:27.383440 (+ 208us) client_negotiation.cc:240] Negotiation successful
0620 06:09:27.383847 (+ 407us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":92,"mutex_wait_us":280,"spinlock_wait_cycles":27520}
I20260620 06:09:27.384122 26972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:09:27.370434 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.102.1:42191 (local address 127.11.102.2:36731)
0620 06:09:27.370769 (+ 335us) server_negotiation.cc:206] Beginning negotiation
0620 06:09:27.370791 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0620 06:09:27.370825 (+ 34us) server_negotiation.cc:407] Connection header received
0620 06:09:27.371483 (+ 658us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:09:27.371488 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:09:27.371906 (+ 418us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:09:27.372187 (+ 281us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:09:27.373503 (+ 1316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:09:27.374436 (+ 933us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:09:27.375724 (+ 1288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:09:27.376163 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:09:27.376320 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:09:27.379674 (+ 3354us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:09:27.379683 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:09:27.379698 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:09:27.379734 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:09:27.381682 (+ 1948us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:09:27.382136 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:09:27.382143 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:09:27.382148 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:09:27.382282 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:09:27.382580 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:09:27.382583 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:09:27.382584 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:09:27.383032 (+ 448us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:09:27.383170 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0620 06:09:27.383558 (+ 388us) server_negotiation.cc:299] Negotiation successful
0620 06:09:27.383884 (+ 326us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":218,"thread_start_us":145,"threads_started":1}
I20260620 06:09:27.384182 26970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:09:27.370091 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.102.3:43859 (local address 127.11.102.1:44665)
0620 06:09:27.370525 (+ 434us) negotiation.cc:107] Waiting for socket to connect
0620 06:09:27.370544 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 06:09:27.370646 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:09:27.372272 (+ 1626us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:09:27.372279 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:09:27.372844 (+ 565us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:09:27.373294 (+ 450us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:09:27.373308 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:09:27.374650 (+ 1342us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:09:27.374653 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:09:27.375546 (+ 893us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:09:27.375555 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:09:27.376348 (+ 793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:09:27.376352 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:09:27.376591 (+ 239us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:09:27.377299 (+ 708us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:09:27.377312 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:09:27.378985 (+ 1673us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:09:27.381327 (+ 2342us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:09:27.381334 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:09:27.381346 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:09:27.381603 (+ 257us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:09:27.382034 (+ 431us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:09:27.382037 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:09:27.382038 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:09:27.382256 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:09:27.382947 (+ 691us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:09:27.382954 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:09:27.383236 (+ 282us) client_negotiation.cc:769] Sending connection context
0620 06:09:27.383442 (+ 206us) client_negotiation.cc:240] Negotiation successful
0620 06:09:27.383941 (+ 499us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":343,"mutex_wait_us":345,"spinlock_wait_cycles":4352,"thread_start_us":80,"threads_started":1}
W20260620 06:09:27.386987 26585 leader_election.cc:341] T b729a999fd6b46d1af62153d893b6656 P d7f871fdb6814adabd0538690e2a65a3 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 0d1347dff7a94072ad8cbe2deef6f05a (127.11.102.2:36731): Not found: Tablet not found: b729a999fd6b46d1af62153d893b6656
W20260620 06:09:27.387370 26584 leader_election.cc:341] T b729a999fd6b46d1af62153d893b6656 P d7f871fdb6814adabd0538690e2a65a3 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 7c99aa7fa7834c89968adb1265fa3603 (127.11.102.3:43859): Not found: Tablet not found: b729a999fd6b46d1af62153d893b6656
I20260620 06:09:27.387439 26584 leader_election.cc:302] T b729a999fd6b46d1af62153d893b6656 P d7f871fdb6814adabd0538690e2a65a3 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d7f871fdb6814adabd0538690e2a65a3; no voters: 0d1347dff7a94072ad8cbe2deef6f05a, 7c99aa7fa7834c89968adb1265fa3603
I20260620 06:09:27.387742 26969 raft_consensus.cc:2726] T b729a999fd6b46d1af62153d893b6656 P d7f871fdb6814adabd0538690e2a65a3 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 06:09:27.876407 26966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:09:27.876086 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.102.62:41475 (local address 127.11.102.1:37405)
0620 06:09:27.876255 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0620 06:09:27.876317 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.102.62:41475: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
I20260620 06:09:27.993129 26973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:09:27.992709 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.102.62:41475 (local address 127.11.102.2:53949)
0620 06:09:27.992973 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0620 06:09:27.993034 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.102.62:41475: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":76,"threads_started":1}
I20260620 06:09:28.223268 26974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:09:28.222897 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.102.62:41475 (local address 127.11.102.3:40919)
0620 06:09:28.223132 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0620 06:09:28.223186 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.102.62:41475: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":169,"thread_start_us":69,"threads_started":1}
I20260620 06:09:28.877743 26975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:09:28.877279 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.102.62:41475 (local address 127.11.102.1:59693)
0620 06:09:28.877589 (+ 310us) negotiation.cc:107] Waiting for socket to connect
0620 06:09:28.877651 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.102.62:41475: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":237,"thread_start_us":80,"threads_started":1}
I20260620 06:09:28.953490 26976 raft_consensus.cc:482] T b729a999fd6b46d1af62153d893b6656 P d7f871fdb6814adabd0538690e2a65a3 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 06:09:28.953619 26976 raft_consensus.cc:504] T b729a999fd6b46d1af62153d893b6656 P d7f871fdb6814adabd0538690e2a65a3 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c99aa7fa7834c89968adb1265fa3603" member_type: VOTER last_known_addr { host: "127.11.102.3" port: 43859 } } peers { permanent_uuid: "0d1347dff7a94072ad8cbe2deef6f05a" member_type: VOTER last_known_addr { host: "127.11.102.2" port: 36731 } } peers { permanent_uuid: "d7f871fdb6814adabd0538690e2a65a3" member_type: VOTER last_known_addr { host: "127.11.102.1" port: 41679 } }
I20260620 06:09:28.954015 26976 leader_election.cc:288] T b729a999fd6b46d1af62153d893b6656 P d7f871fdb6814adabd0538690e2a65a3 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 7c99aa7fa7834c89968adb1265fa3603 (127.11.102.3:43859), 0d1347dff7a94072ad8cbe2deef6f05a (127.11.102.2:36731)
W20260620 06:09:28.954876 26585 leader_election.cc:341] T b729a999fd6b46d1af62153d893b6656 P d7f871fdb6814adabd0538690e2a65a3 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 0d1347dff7a94072ad8cbe2deef6f05a (127.11.102.2:36731): Not found: Tablet not found: b729a999fd6b46d1af62153d893b6656
W20260620 06:09:28.954991 26584 leader_election.cc:341] T b729a999fd6b46d1af62153d893b6656 P d7f871fdb6814adabd0538690e2a65a3 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 7c99aa7fa7834c89968adb1265fa3603 (127.11.102.3:43859): Not found: Tablet not found: b729a999fd6b46d1af62153d893b6656
I20260620 06:09:28.955108 26584 leader_election.cc:302] T b729a999fd6b46d1af62153d893b6656 P d7f871fdb6814adabd0538690e2a65a3 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d7f871fdb6814adabd0538690e2a65a3; no voters: 0d1347dff7a94072ad8cbe2deef6f05a, 7c99aa7fa7834c89968adb1265fa3603
I20260620 06:09:28.955317 26976 raft_consensus.cc:2726] T b729a999fd6b46d1af62153d893b6656 P d7f871fdb6814adabd0538690e2a65a3 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 06:09:28.994354 26977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:09:28.993928 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.102.62:41475 (local address 127.11.102.2:47125)
0620 06:09:28.994198 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0620 06:09:28.994254 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.102.62:41475: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":201,"thread_start_us":75,"threads_started":1}
I20260620 06:09:29.224459 26978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:09:29.224058 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.102.62:41475 (local address 127.11.102.3:44917)
0620 06:09:29.224298 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0620 06:09:29.224354 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.102.62:41475: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":169,"thread_start_us":72,"threads_started":1}
I20260620 06:09:29.879040 26979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:09:29.878584 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.102.62:41475 (local address 127.11.102.1:60465)
0620 06:09:29.878874 (+ 290us) negotiation.cc:107] Waiting for socket to connect
0620 06:09:29.878946 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.102.62:41475: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":206,"thread_start_us":93,"threads_started":1}
I20260620 06:09:29.995530 26980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:09:29.995100 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.102.62:41475 (local address 127.11.102.2:49591)
0620 06:09:29.995366 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0620 06:09:29.995419 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.102.62:41475: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":81,"threads_started":1}
I20260620 06:09:30.225555 26981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:09:30.225162 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.102.62:41475 (local address 127.11.102.3:50981)
0620 06:09:30.225413 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0620 06:09:30.225469 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.102.62:41475: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":83,"threads_started":1}
I20260620 06:09:30.880477 26982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:09:30.880016 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.102.62:41475 (local address 127.11.102.1:44493)
0620 06:09:30.880320 (+ 304us) negotiation.cc:107] Waiting for socket to connect
0620 06:09:30.880383 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.102.62:41475: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":227,"thread_start_us":74,"threads_started":1}
I20260620 06:09:30.996763 26983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:09:30.996299 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.102.62:41475 (local address 127.11.102.2:53055)
0620 06:09:30.996613 (+ 314us) negotiation.cc:107] Waiting for socket to connect
0620 06:09:30.996670 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.102.62:41475: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":234,"thread_start_us":96,"threads_started":1}
I20260620 06:09:31.226846 26984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:09:31.226375 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.102.62:41475 (local address 127.11.102.3:56785)
0620 06:09:31.226692 (+ 317us) negotiation.cc:107] Waiting for socket to connect
0620 06:09:31.226753 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.102.62:41475: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":244,"thread_start_us":92,"threads_started":1}
I20260620 06:09:31.881763 26985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:09:31.881287 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.102.62:41475 (local address 127.11.102.1:36897)
0620 06:09:31.881606 (+ 319us) negotiation.cc:107] Waiting for socket to connect
0620 06:09:31.881669 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.102.62:41475: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":244,"thread_start_us":112,"threads_started":1}
I20260620 06:09:31.997987 26986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:09:31.997573 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.102.62:41475 (local address 127.11.102.2:42649)
0620 06:09:31.997831 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0620 06:09:31.997890 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.102.62:41475: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":78,"threads_started":1}
I20260620 06:09:32.095994 26987 raft_consensus.cc:482] T b729a999fd6b46d1af62153d893b6656 P d7f871fdb6814adabd0538690e2a65a3 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 06:09:32.096161 26987 raft_consensus.cc:504] T b729a999fd6b46d1af62153d893b6656 P d7f871fdb6814adabd0538690e2a65a3 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c99aa7fa7834c89968adb1265fa3603" member_type: VOTER last_known_addr { host: "127.11.102.3" port: 43859 } } peers { permanent_uuid: "0d1347dff7a94072ad8cbe2deef6f05a" member_type: VOTER last_known_addr { host: "127.11.102.2" port: 36731 } } peers { permanent_uuid: "d7f871fdb6814adabd0538690e2a65a3" member_type: VOTER last_known_addr { host: "127.11.102.1" port: 41679 } }
I20260620 06:09:32.096640 26987 leader_election.cc:288] T b729a999fd6b46d1af62153d893b6656 P d7f871fdb6814adabd0538690e2a65a3 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 7c99aa7fa7834c89968adb1265fa3603 (127.11.102.3:43859), 0d1347dff7a94072ad8cbe2deef6f05a (127.11.102.2:36731)
W20260620 06:09:32.097510 26585 leader_election.cc:341] T b729a999fd6b46d1af62153d893b6656 P d7f871fdb6814adabd0538690e2a65a3 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 0d1347dff7a94072ad8cbe2deef6f05a (127.11.102.2:36731): Not found: Tablet not found: b729a999fd6b46d1af62153d893b6656
W20260620 06:09:32.097693 26584 leader_election.cc:341] T b729a999fd6b46d1af62153d893b6656 P d7f871fdb6814adabd0538690e2a65a3 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 7c99aa7fa7834c89968adb1265fa3603 (127.11.102.3:43859): Not found: Tablet not found: b729a999fd6b46d1af62153d893b6656
I20260620 06:09:32.097779 26584 leader_election.cc:302] T b729a999fd6b46d1af62153d893b6656 P d7f871fdb6814adabd0538690e2a65a3 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d7f871fdb6814adabd0538690e2a65a3; no voters: 0d1347dff7a94072ad8cbe2deef6f05a, 7c99aa7fa7834c89968adb1265fa3603
I20260620 06:09:32.097954 26987 raft_consensus.cc:2726] T b729a999fd6b46d1af62153d893b6656 P d7f871fdb6814adabd0538690e2a65a3 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 06:09:32.228066 26988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:09:32.227684 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.102.62:41475 (local address 127.11.102.3:36929)
0620 06:09:32.227925 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0620 06:09:32.227980 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.102.62:41475: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":162,"thread_start_us":68,"threads_started":1}
I20260620 06:09:32.882977 26989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:09:32.882569 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.102.62:41475 (local address 127.11.102.1:48995)
0620 06:09:32.882814 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0620 06:09:32.882883 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.102.62:41475: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":70,"threads_started":1}
I20260620 06:09:32.999396 26990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:09:32.998893 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.102.62:41475 (local address 127.11.102.2:34869)
0620 06:09:32.999241 (+ 348us) negotiation.cc:107] Waiting for socket to connect
0620 06:09:32.999302 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.102.62:41475: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":271,"thread_start_us":91,"threads_started":1}
I20260620 06:09:33.229262 26991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:09:33.228803 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.102.62:41475 (local address 127.11.102.3:34539)
0620 06:09:33.229115 (+ 312us) negotiation.cc:107] Waiting for socket to connect
0620 06:09:33.229171 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.102.62:41475: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":231,"thread_start_us":98,"threads_started":1}
I20260620 06:09:33.884177 26992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:09:33.883770 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.102.62:41475 (local address 127.11.102.1:51597)
0620 06:09:33.884021 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0620 06:09:33.884072 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.102.62:41475: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":73,"threads_started":1}
I20260620 06:09:34.000769 26993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:09:34.000208 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.102.62:41475 (local address 127.11.102.2:43769)
0620 06:09:34.000581 (+ 373us) negotiation.cc:107] Waiting for socket to connect
0620 06:09:34.000653 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.102.62:41475: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":249,"thread_start_us":102,"threads_started":1}
I20260620 06:09:34.230337 26994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:09:34.229952 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.102.62:41475 (local address 127.11.102.3:34153)
0620 06:09:34.230198 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0620 06:09:34.230250 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.102.62:41475: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":71,"threads_started":1}
I20260620 06:09:34.873414 26995 raft_consensus.cc:482] T b729a999fd6b46d1af62153d893b6656 P d7f871fdb6814adabd0538690e2a65a3 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 06:09:34.873540 26995 raft_consensus.cc:504] T b729a999fd6b46d1af62153d893b6656 P d7f871fdb6814adabd0538690e2a65a3 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c99aa7fa7834c89968adb1265fa3603" member_type: VOTER last_known_addr { host: "127.11.102.3" port: 43859 } } peers { permanent_uuid: "0d1347dff7a94072ad8cbe2deef6f05a" member_type: VOTER last_known_addr { host: "127.11.102.2" port: 36731 } } peers { permanent_uuid: "d7f871fdb6814adabd0538690e2a65a3" member_type: VOTER last_known_addr { host: "127.11.102.1" port: 41679 } }
I20260620 06:09:34.873893 26995 leader_election.cc:288] T b729a999fd6b46d1af62153d893b6656 P d7f871fdb6814adabd0538690e2a65a3 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 7c99aa7fa7834c89968adb1265fa3603 (127.11.102.3:43859), 0d1347dff7a94072ad8cbe2deef6f05a (127.11.102.2:36731)
W20260620 06:09:34.874716 26584 leader_election.cc:341] T b729a999fd6b46d1af62153d893b6656 P d7f871fdb6814adabd0538690e2a65a3 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 7c99aa7fa7834c89968adb1265fa3603 (127.11.102.3:43859): Not found: Tablet not found: b729a999fd6b46d1af62153d893b6656
W20260620 06:09:34.874807 26585 leader_election.cc:341] T b729a999fd6b46d1af62153d893b6656 P d7f871fdb6814adabd0538690e2a65a3 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 0d1347dff7a94072ad8cbe2deef6f05a (127.11.102.2:36731): Not found: Tablet not found: b729a999fd6b46d1af62153d893b6656
I20260620 06:09:34.874871 26585 leader_election.cc:302] T b729a999fd6b46d1af62153d893b6656 P d7f871fdb6814adabd0538690e2a65a3 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d7f871fdb6814adabd0538690e2a65a3; no voters: 0d1347dff7a94072ad8cbe2deef6f05a, 7c99aa7fa7834c89968adb1265fa3603
I20260620 06:09:34.875037 26995 raft_consensus.cc:2726] T b729a999fd6b46d1af62153d893b6656 P d7f871fdb6814adabd0538690e2a65a3 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 06:09:34.885267 26996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:09:34.884884 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.102.62:41475 (local address 127.11.102.1:53831)
0620 06:09:34.885132 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0620 06:09:34.885181 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.102.62:41475: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":161,"thread_start_us":95,"threads_started":1}
I20260620 06:09:35.001935 26997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:09:35.001525 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.102.62:41475 (local address 127.11.102.2:46053)
0620 06:09:35.001790 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0620 06:09:35.001844 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.102.62:41475: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":76,"threads_started":1}
I20260620 06:09:35.231482 26998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:09:35.231085 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.102.62:41475 (local address 127.11.102.3:38091)
0620 06:09:35.231337 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0620 06:09:35.231392 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.102.62:41475: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":80,"threads_started":1}
I20260620 06:09:35.886736 26999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:09:35.886259 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.102.62:41475 (local address 127.11.102.1:57075)
0620 06:09:35.886553 (+ 294us) negotiation.cc:107] Waiting for socket to connect
0620 06:09:35.886643 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.102.62:41475: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":86,"threads_started":1}
I20260620 06:09:36.003126 27000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:09:36.002711 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.102.62:41475 (local address 127.11.102.2:49743)
0620 06:09:36.002975 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0620 06:09:36.003034 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.102.62:41475: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":74,"threads_started":1}
I20260620 06:09:36.232703 27001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:09:36.232305 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.102.62:41475 (local address 127.11.102.3:37339)
0620 06:09:36.232554 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0620 06:09:36.232610 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.102.62:41475: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":83,"threads_started":1}
I20260620 06:09:36.825744 27002 raft_consensus.cc:482] T b729a999fd6b46d1af62153d893b6656 P d7f871fdb6814adabd0538690e2a65a3 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 06:09:36.825914 27002 raft_consensus.cc:504] T b729a999fd6b46d1af62153d893b6656 P d7f871fdb6814adabd0538690e2a65a3 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c99aa7fa7834c89968adb1265fa3603" member_type: VOTER last_known_addr { host: "127.11.102.3" port: 43859 } } peers { permanent_uuid: "0d1347dff7a94072ad8cbe2deef6f05a" member_type: VOTER last_known_addr { host: "127.11.102.2" port: 36731 } } peers { permanent_uuid: "d7f871fdb6814adabd0538690e2a65a3" member_type: VOTER last_known_addr { host: "127.11.102.1" port: 41679 } }
I20260620 06:09:36.826342 27002 leader_election.cc:288] T b729a999fd6b46d1af62153d893b6656 P d7f871fdb6814adabd0538690e2a65a3 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 7c99aa7fa7834c89968adb1265fa3603 (127.11.102.3:43859), 0d1347dff7a94072ad8cbe2deef6f05a (127.11.102.2:36731)
W20260620 06:09:36.827247 26584 leader_election.cc:341] T b729a999fd6b46d1af62153d893b6656 P d7f871fdb6814adabd0538690e2a65a3 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 7c99aa7fa7834c89968adb1265fa3603 (127.11.102.3:43859): Not found: Tablet not found: b729a999fd6b46d1af62153d893b6656
W20260620 06:09:36.827348 26585 leader_election.cc:341] T b729a999fd6b46d1af62153d893b6656 P d7f871fdb6814adabd0538690e2a65a3 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 0d1347dff7a94072ad8cbe2deef6f05a (127.11.102.2:36731): Not found: Tablet not found: b729a999fd6b46d1af62153d893b6656
I20260620 06:09:36.827414 26584 leader_election.cc:302] T b729a999fd6b46d1af62153d893b6656 P d7f871fdb6814adabd0538690e2a65a3 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d7f871fdb6814adabd0538690e2a65a3; no voters: 0d1347dff7a94072ad8cbe2deef6f05a, 7c99aa7fa7834c89968adb1265fa3603
I20260620 06:09:36.827592 27002 raft_consensus.cc:2726] T b729a999fd6b46d1af62153d893b6656 P d7f871fdb6814adabd0538690e2a65a3 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 06:09:36.888018 27003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:09:36.887480 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.102.62:41475 (local address 127.11.102.1:52033)
0620 06:09:36.887804 (+ 324us) negotiation.cc:107] Waiting for socket to connect
0620 06:09:36.887901 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.102.62:41475: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":232,"thread_start_us":81,"threads_started":1}
I20260620 06:09:37.004273 27004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:09:37.003900 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.102.62:41475 (local address 127.11.102.2:41083)
0620 06:09:37.004124 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0620 06:09:37.004175 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.102.62:41475: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":68,"threads_started":1}
I20260620 06:09:37.233954 27005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:09:37.233465 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.102.62:41475 (local address 127.11.102.3:53847)
0620 06:09:37.233795 (+ 330us) negotiation.cc:107] Waiting for socket to connect
0620 06:09:37.233851 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.102.62:41475: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":259,"thread_start_us":89,"threads_started":1}
I20260620 06:09:37.889287 27006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:09:37.888822 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.102.62:41475 (local address 127.11.102.1:56541)
0620 06:09:37.889112 (+ 290us) negotiation.cc:107] Waiting for socket to connect
0620 06:09:37.889170 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.102.62:41475: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":214,"thread_start_us":80,"threads_started":1}
I20260620 06:09:38.005457 27007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:09:38.005000 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.102.62:41475 (local address 127.11.102.2:41085)
0620 06:09:38.005298 (+ 298us) negotiation.cc:107] Waiting for socket to connect
0620 06:09:38.005362 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.102.62:41475: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":72,"threads_started":1}
I20260620 06:09:38.235174 27008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:09:38.234797 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.102.62:41475 (local address 127.11.102.3:33535)
0620 06:09:38.235028 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0620 06:09:38.235084 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.102.62:41475: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":163,"thread_start_us":70,"threads_started":1}
I20260620 06:09:38.890497 27009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:09:38.890074 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.102.62:41475 (local address 127.11.102.1:53091)
0620 06:09:38.890352 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0620 06:09:38.890407 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.102.62:41475: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":209,"thread_start_us":125,"threads_started":1}
I20260620 06:09:39.006665 27010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:09:39.006239 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.102.62:41475 (local address 127.11.102.2:59183)
0620 06:09:39.006512 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0620 06:09:39.006570 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.102.62:41475: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":202,"thread_start_us":83,"threads_started":1}
I20260620 06:09:39.236341 27011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:09:39.235930 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.102.62:41475 (local address 127.11.102.3:59059)
0620 06:09:39.236192 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0620 06:09:39.236249 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.102.62:41475: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":173,"thread_start_us":69,"threads_started":1}
I20260620 06:09:39.891781 27012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:09:39.891353 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.102.62:41475 (local address 127.11.102.1:60627)
0620 06:09:39.891580 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0620 06:09:39.891688 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.102.62:41475: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":164,"thread_start_us":69,"threads_started":1}
I20260620 06:09:40.007942 27013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:09:40.007469 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.102.62:41475 (local address 127.11.102.2:37105)
0620 06:09:40.007790 (+ 321us) negotiation.cc:107] Waiting for socket to connect
0620 06:09:40.007843 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.102.62:41475: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":253,"thread_start_us":76,"threads_started":1}
I20260620 06:09:40.237543 27014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:09:40.237066 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.102.62:41475 (local address 127.11.102.3:47879)
0620 06:09:40.237394 (+ 328us) negotiation.cc:107] Waiting for socket to connect
0620 06:09:40.237449 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.102.62:41475: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":248,"thread_start_us":116,"threads_started":1}
I20260620 06:09:40.893103 27015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:09:40.892710 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.102.62:41475 (local address 127.11.102.1:56515)
0620 06:09:40.892958 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0620 06:09:40.893012 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.102.62:41475: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":84,"threads_started":1}
I20260620 06:09:41.009382 27016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:09:41.008923 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.102.62:41475 (local address 127.11.102.2:58163)
0620 06:09:41.009205 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0620 06:09:41.009273 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.102.62:41475: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":85,"threads_started":1}
I20260620 06:09:41.224633 11672 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9HMp43/build/debug/bin/kudu with pid 26532
I20260620 06:09:41.230268 11672 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9HMp43/build/debug/bin/kudu with pid 26699
I20260620 06:09:41.235548 11672 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9HMp43/build/debug/bin/kudu with pid 26833
2026-06-20T06:09:41Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (19094 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935603120653-11672-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 06:09:41 dist-test-slave-9hm5 krb5kdc[27019](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 06:09:41 dist-test-slave-9hm5 krb5kdc[27019](info): set up 2 sockets
Jun 20 06:09:41 dist-test-slave-9hm5 krb5kdc[27019](info): commencing operation
krb5kdc: starting...
W20260620 06:09:42.279838 11672 mini_kdc.cc:121] Time spent starting KDC: real 1.023s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935603120653-11672-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935603120653-11672-0/krb5kdc/test-user.keytab.
Jun 20 06:09:42 dist-test-slave-9hm5 krb5kdc[27019](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935782, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T06:09:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T06:09:42Z Disabled control of system clock
WARNING: no policy specified for kudu/127.11.102.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.102.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.102.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935603120653-11672-0/krb5kdc/kudu_127.11.102.62.keytab.
Entry for principal kudu/127.11.102.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935603120653-11672-0/krb5kdc/kudu_127.11.102.62.keytab.
I20260620 06:09:42.400527 11672 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9HMp43/build/debug/bin/kudu
/tmp/dist-test-task9HMp43/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935603120653-11672-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935603120653-11672-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935603120653-11672-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935603120653-11672-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.102.62:41831
--webserver_interface=127.11.102.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935603120653-11672-0/krb5kdc/kudu_127.11.102.62.keytab
--principal=kudu/127.11.102.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.102.20:38439
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.102.62:41831
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935603120653-11672-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935603120653-11672-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935603120653-11672-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 06:09:42.510041 27032 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:09:42.510313 27032 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:09:42.510354 27032 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:09:42.514312 27032 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260620 06:09:42.514360 27032 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260620 06:09:42.514376 27032 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260620 06:09:42.514390 27032 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 06:09:42.514408 27032 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:09:42.514425 27032 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:09:42.514441 27032 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 06:09:42.514465 27032 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 06:09:42.519783 27032 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.102.20:38439
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935603120653-11672-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935603120653-11672-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.11.102.62:41831
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935603120653-11672-0/krb5kdc/kudu_127.11.102.62.keytab
--principal=kudu/127.11.102.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.102.62:41831
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935603120653-11672-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.102.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27032
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935603120653-11672-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 05:28:48 UTC on ccycloud.quasar-fixhms.root.comops.site
build id 4457020
I20260620 06:09:42.520773 27032 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:09:42.521750 27032 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 06:09:42.530413 27037 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:09:42.530454 27038 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:09:42.530562 27040 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:09:42.530579 27032 server_base.cc:1034] running on GCE node
I20260620 06:09:42.531325 27032 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:09:42.532404 27032 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:09:42.533605 27032 hybrid_clock.cc:648] HybridClock initialized: now 1781935782533578 us; error 37 us; skew 500 ppm
Jun 20 06:09:42 dist-test-slave-9hm5 krb5kdc[27019](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935782, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.102.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 06:09:42.536458 27032 init.cc:377] Logged in from keytab as kudu/127.11.102.62@KRBTEST.COM (short username kudu)
I20260620 06:09:42.537384 27032 webserver.cc:460] Webserver started at http://127.11.102.62:41901/ using document root <none> and password file <none>
I20260620 06:09:42.537973 27032 fs_manager.cc:359] Metadata directory not provided
I20260620 06:09:42.538069 27032 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:09:42.538303 27032 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 06:09:42.540477 27032 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935603120653-11672-0/minicluster-data/master-0/data/instance:
uuid: "08cf14c4a270431082d354a7a84ab2fd"
format_stamp: "Formatted at 2026-06-20 06:09:42 on dist-test-slave-9hm5"
server_key: "30abf25c0b3089745fb551a08427a3b0"
server_key_iv: "5d2f4a95c2affcce55c4ce2dcb4b6639"
server_key_version: "encryptionkey@0"
I20260620 06:09:42.541040 27032 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935603120653-11672-0/minicluster-data/master-0/wal/instance:
uuid: "08cf14c4a270431082d354a7a84ab2fd"
format_stamp: "Formatted at 2026-06-20 06:09:42 on dist-test-slave-9hm5"
server_key: "30abf25c0b3089745fb551a08427a3b0"
server_key_iv: "5d2f4a95c2affcce55c4ce2dcb4b6639"
server_key_version: "encryptionkey@0"
I20260620 06:09:42.544721 27032 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 06:09:42.547034 27047 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:09:42.548295 27032 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 06:09:42.548432 27032 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935603120653-11672-0/minicluster-data/master-0/data,/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935603120653-11672-0/minicluster-data/master-0/wal
uuid: "08cf14c4a270431082d354a7a84ab2fd"
format_stamp: "Formatted at 2026-06-20 06:09:42 on dist-test-slave-9hm5"
server_key: "30abf25c0b3089745fb551a08427a3b0"
server_key_iv: "5d2f4a95c2affcce55c4ce2dcb4b6639"
server_key_version: "encryptionkey@0"
I20260620 06:09:42.548542 27032 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935603120653-11672-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935603120653-11672-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935603120653-11672-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:09:42.572161 27032 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:09:42.575466 27032 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:09:42.575703 27032 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:09:42.582963 27032 rpc_server.cc:307] RPC server started. Bound to: 127.11.102.62:41831
I20260620 06:09:42.583261 27099 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.102.62:41831 every 8 connection(s)
I20260620 06:09:42.583971 27032 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935603120653-11672-0/minicluster-data/master-0/data/info.pb
I20260620 06:09:42.586781 27100 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 06:09:42.586769 11672 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9HMp43/build/debug/bin/kudu as pid 27032
I20260620 06:09:42.586895 11672 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935603120653-11672-0/minicluster-data/master-0/wal/instance
I20260620 06:09:42.587224 11672 external_mini_cluster.cc:1428] Setting key 1a81d876211aa35e759f7b8aae0d899a
I20260620 06:09:42.593619 27100 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 08cf14c4a270431082d354a7a84ab2fd: Bootstrap starting.
I20260620 06:09:42.596329 27100 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 08cf14c4a270431082d354a7a84ab2fd: Neither blocks nor log segments found. Creating new log.
Jun 20 06:09:42 dist-test-slave-9hm5 krb5kdc[27019](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935782, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.102.62@KRBTEST.COM
I20260620 06:09:42.597149 27100 log.cc:824] T 00000000000000000000000000000000 P 08cf14c4a270431082d354a7a84ab2fd: Log is configured to *not* fsync() on all Append() calls
I20260620 06:09:42.599377 27100 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 08cf14c4a270431082d354a7a84ab2fd: No bootstrap required, opened a new log
I20260620 06:09:42.602066 27103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:09:42.588666 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33370 (local address 127.11.102.62:41831)
0620 06:09:42.589079 (+ 413us) server_negotiation.cc:206] Beginning negotiation
0620 06:09:42.589094 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 06:09:42.589129 (+ 35us) server_negotiation.cc:407] Connection header received
0620 06:09:42.589987 (+ 858us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:09:42.590006 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:09:42.590350 (+ 344us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:09:42.590631 (+ 281us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:09:42.591513 (+ 882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:09:42.592543 (+ 1030us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:09:42.593663 (+ 1120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:09:42.594190 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:09:42.594306 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:09:42.597031 (+ 2725us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:09:42.597037 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:09:42.597052 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:09:42.597082 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:09:42.598963 (+ 1881us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:09:42.599407 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:09:42.599414 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:09:42.599421 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:09:42.599551 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:09:42.599873 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:09:42.599876 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:09:42.599877 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:09:42.600355 (+ 478us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:09:42.600549 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0620 06:09:42.600783 (+ 234us) server_negotiation.cc:299] Negotiation successful
0620 06:09:42.601065 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":265,"thread_start_us":141,"threads_started":1}
I20260620 06:09:42.603574 27100 raft_consensus.cc:348] T 00000000000000000000000000000000 P 08cf14c4a270431082d354a7a84ab2fd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "08cf14c4a270431082d354a7a84ab2fd" member_type: VOTER last_known_addr { host: "127.11.102.62" port: 41831 } }
I20260620 06:09:42.603852 27100 raft_consensus.cc:374] T 00000000000000000000000000000000 P 08cf14c4a270431082d354a7a84ab2fd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 06:09:42.603899 27100 raft_consensus.cc:729] T 00000000000000000000000000000000 P 08cf14c4a270431082d354a7a84ab2fd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 08cf14c4a270431082d354a7a84ab2fd, State: Initialized, Role: FOLLOWER
I20260620 06:09:42.604324 27100 consensus_queue.cc:260] T 00000000000000000000000000000000 P 08cf14c4a270431082d354a7a84ab2fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "08cf14c4a270431082d354a7a84ab2fd" member_type: VOTER last_known_addr { host: "127.11.102.62" port: 41831 } }
I20260620 06:09:42.604480 27100 raft_consensus.cc:388] T 00000000000000000000000000000000 P 08cf14c4a270431082d354a7a84ab2fd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 06:09:42.604544 27100 raft_consensus.cc:482] T 00000000000000000000000000000000 P 08cf14c4a270431082d354a7a84ab2fd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 06:09:42.604662 27100 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 08cf14c4a270431082d354a7a84ab2fd [term 0 FOLLOWER]: Advancing to term 1
I20260620 06:09:42.605538 27100 raft_consensus.cc:504] T 00000000000000000000000000000000 P 08cf14c4a270431082d354a7a84ab2fd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "08cf14c4a270431082d354a7a84ab2fd" member_type: VOTER last_known_addr { host: "127.11.102.62" port: 41831 } }
I20260620 06:09:42.605862 27100 leader_election.cc:302] T 00000000000000000000000000000000 P 08cf14c4a270431082d354a7a84ab2fd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 08cf14c4a270431082d354a7a84ab2fd; no voters:
I20260620 06:09:42.606153 27100 leader_election.cc:288] T 00000000000000000000000000000000 P 08cf14c4a270431082d354a7a84ab2fd [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 06:09:42.606300 27105 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 08cf14c4a270431082d354a7a84ab2fd [term 1 FOLLOWER]: Leader election won for term 1
I20260620 06:09:42.606547 27105 raft_consensus.cc:686] T 00000000000000000000000000000000 P 08cf14c4a270431082d354a7a84ab2fd [term 1 LEADER]: Becoming Leader. State: Replica: 08cf14c4a270431082d354a7a84ab2fd, State: Running, Role: LEADER
I20260620 06:09:42.606879 27105 consensus_queue.cc:237] T 00000000000000000000000000000000 P 08cf14c4a270431082d354a7a84ab2fd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "08cf14c4a270431082d354a7a84ab2fd" member_type: VOTER last_known_addr { host: "127.11.102.62" port: 41831 } }
I20260620 06:09:42.607285 27100 sys_catalog.cc:564] T 00000000000000000000000000000000 P 08cf14c4a270431082d354a7a84ab2fd [sys.catalog]: configured and running, proceeding with master startup.
I20260620 06:09:42.608534 27106 sys_catalog.cc:455] T 00000000000000000000000000000000 P 08cf14c4a270431082d354a7a84ab2fd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "08cf14c4a270431082d354a7a84ab2fd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "08cf14c4a270431082d354a7a84ab2fd" member_type: VOTER last_known_addr { host: "127.11.102.62" port: 41831 } } }
I20260620 06:09:42.608671 27106 sys_catalog.cc:458] T 00000000000000000000000000000000 P 08cf14c4a270431082d354a7a84ab2fd [sys.catalog]: This master's current role is: LEADER
I20260620 06:09:42.609073 27114 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 06:09:42.609516 27107 sys_catalog.cc:455] T 00000000000000000000000000000000 P 08cf14c4a270431082d354a7a84ab2fd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 08cf14c4a270431082d354a7a84ab2fd. Latest consensus state: current_term: 1 leader_uuid: "08cf14c4a270431082d354a7a84ab2fd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "08cf14c4a270431082d354a7a84ab2fd" member_type: VOTER last_known_addr { host: "127.11.102.62" port: 41831 } } }
I20260620 06:09:42.609733 27107 sys_catalog.cc:458] T 00000000000000000000000000000000 P 08cf14c4a270431082d354a7a84ab2fd [sys.catalog]: This master's current role is: LEADER
I20260620 06:09:42.612604 27114 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 06:09:42.617884 27114 catalog_manager.cc:1347] Generated new cluster ID: c5bf0008db3641f5aa12f375b47b3a94
I20260620 06:09:42.617949 27114 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 06:09:42.647212 27114 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 06:09:42.648607 27114 catalog_manager.cc:1504] Loading token signing keys...
I20260620 06:09:42.663998 27114 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 08cf14c4a270431082d354a7a84ab2fd: Generated new TSK 0
I20260620 06:09:42.664752 27114 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.11.102.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.102.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.102.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935603120653-11672-0/krb5kdc/kudu_127.11.102.1.keytab.
Entry for principal kudu/127.11.102.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935603120653-11672-0/krb5kdc/kudu_127.11.102.1.keytab.
I20260620 06:09:42.698401 11672 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9HMp43/build/debug/bin/kudu
/tmp/dist-test-task9HMp43/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935603120653-11672-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935603120653-11672-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935603120653-11672-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935603120653-11672-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.102.1:0
--local_ip_for_outbound_sockets=127.11.102.1
--webserver_interface=127.11.102.1
--webserver_port=0
--tserver_master_addrs=127.11.102.62:41831
--keytab_file=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935603120653-11672-0/krb5kdc/kudu_127.11.102.1.keytab
--principal=kudu/127.11.102.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.102.20:38439
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935603120653-11672-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935603120653-11672-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935603120653-11672-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 06:09:42.807387 27126 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:09:42.807776 27126 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:09:42.807827 27126 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:09:42.812007 27126 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:09:42.812062 27126 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:09:42.812125 27126 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.102.1
I20260620 06:09:42.817239 27126 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.102.20:38439
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935603120653-11672-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935603120653-11672-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935603120653-11672-0/krb5kdc/kudu_127.11.102.1.keytab
--principal=kudu/127.11.102.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.102.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935603120653-11672-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.11.102.1
--webserver_port=0
--tserver_master_addrs=127.11.102.62:41831
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27126
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.102.1
--log_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935603120653-11672-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 05:28:48 UTC on ccycloud.quasar-fixhms.root.comops.site
build id 4457020
I20260620 06:09:42.818200 27126 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:09:42.819111 27126 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 06:09:42.828281 27132 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:09:42.828253 27134 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:09:42.828328 27126 server_base.cc:1034] running on GCE node
W20260620 06:09:42.828477 27131 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:09:42.828881 27126 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:09:42.829486 27126 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:09:42.830696 27126 hybrid_clock.cc:648] HybridClock initialized: now 1781935782830668 us; error 38 us; skew 500 ppm
Jun 20 06:09:42 dist-test-slave-9hm5 krb5kdc[27019](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935782, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.102.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 06:09:42.833458 27126 init.cc:377] Logged in from keytab as kudu/127.11.102.1@KRBTEST.COM (short username kudu)
I20260620 06:09:42.834348 27126 webserver.cc:460] Webserver started at http://127.11.102.1:34149/ using document root <none> and password file <none>
I20260620 06:09:42.834887 27126 fs_manager.cc:359] Metadata directory not provided
I20260620 06:09:42.834973 27126 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:09:42.835222 27126 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 06:09:42.837232 27126 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935603120653-11672-0/minicluster-data/ts-0/data/instance:
uuid: "3fd44bea82cc4e89bfc9ada83a1fd302"
format_stamp: "Formatted at 2026-06-20 06:09:42 on dist-test-slave-9hm5"
server_key: "87f03c04be86da2aa07d685cba61f524"
server_key_iv: "0050b961ddf0b580701336a7fd81ae80"
server_key_version: "encryptionkey@0"
I20260620 06:09:42.837760 27126 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935603120653-11672-0/minicluster-data/ts-0/wal/instance:
uuid: "3fd44bea82cc4e89bfc9ada83a1fd302"
format_stamp: "Formatted at 2026-06-20 06:09:42 on dist-test-slave-9hm5"
server_key: "87f03c04be86da2aa07d685cba61f524"
server_key_iv: "0050b961ddf0b580701336a7fd81ae80"
server_key_version: "encryptionkey@0"
I20260620 06:09:42.841373 27126 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.005s
I20260620 06:09:42.843685 27141 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:09:42.844842 27126 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.002s
I20260620 06:09:42.844967 27126 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935603120653-11672-0/minicluster-data/ts-0/data,/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935603120653-11672-0/minicluster-data/ts-0/wal
uuid: "3fd44bea82cc4e89bfc9ada83a1fd302"
format_stamp: "Formatted at 2026-06-20 06:09:42 on dist-test-slave-9hm5"
server_key: "87f03c04be86da2aa07d685cba61f524"
server_key_iv: "0050b961ddf0b580701336a7fd81ae80"
server_key_version: "encryptionkey@0"
I20260620 06:09:42.845104 27126 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935603120653-11672-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935603120653-11672-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935603120653-11672-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:09:42.866500 27126 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:09:42.869388 27126 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:09:42.869586 27126 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:09:42.870167 27126 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 06:09:42.870955 27126 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 06:09:42.871032 27126 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 06:09:42.871124 27126 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 06:09:42.871174 27126 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 06:09:42.879814 27126 rpc_server.cc:307] RPC server started. Bound to: 127.11.102.1:37171
I20260620 06:09:42.879848 27254 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.102.1:37171 every 8 connection(s)
I20260620 06:09:42.880648 27126 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935603120653-11672-0/minicluster-data/ts-0/data/info.pb
I20260620 06:09:42.884688 11672 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9HMp43/build/debug/bin/kudu as pid 27126
I20260620 06:09:42.884821 11672 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935603120653-11672-0/minicluster-data/ts-0/wal/instance
I20260620 06:09:42.885128 11672 external_mini_cluster.cc:1428] Setting key adda162e94acf0008a574276904bdf0e
Jun 20 06:09:42 dist-test-slave-9hm5 krb5kdc[27019](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935782, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.102.1@KRBTEST.COM for kudu/127.11.102.62@KRBTEST.COM
I20260620 06:09:42.894889 27103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:09:42.882301 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.102.1:41443 (local address 127.11.102.62:41831)
0620 06:09:42.882430 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0620 06:09:42.882434 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 06:09:42.883166 (+ 732us) server_negotiation.cc:407] Connection header received
0620 06:09:42.884490 (+ 1324us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:09:42.884494 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:09:42.884562 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:09:42.884683 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:09:42.886033 (+ 1350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:09:42.886680 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:09:42.887847 (+ 1167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:09:42.888094 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:09:42.888226 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:09:42.890974 (+ 2748us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:09:42.890980 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:09:42.890985 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:09:42.891019 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:09:42.892804 (+ 1785us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:09:42.893315 (+ 511us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:09:42.893320 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:09:42.893323 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:09:42.893420 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:09:42.893749 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:09:42.893752 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:09:42.893754 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:09:42.893995 (+ 241us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:09:42.894101 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0620 06:09:42.894575 (+ 474us) server_negotiation.cc:299] Negotiation successful
0620 06:09:42.894723 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
WARNING: no policy specified for kudu/127.11.102.2@KRBTEST.COM; defaulting to no policy
I20260620 06:09:42.895654 27257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:09:42.882604 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.102.62:41831 (local address 127.11.102.1:41443)
0620 06:09:42.883008 (+ 404us) negotiation.cc:107] Waiting for socket to connect
0620 06:09:42.883049 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0620 06:09:42.884220 (+ 1171us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:09:42.884887 (+ 667us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:09:42.884898 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:09:42.885310 (+ 412us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:09:42.885878 (+ 568us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:09:42.885893 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:09:42.886807 (+ 914us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:09:42.886811 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:09:42.887728 (+ 917us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:09:42.887738 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:09:42.888262 (+ 524us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:09:42.888267 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:09:42.888432 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:09:42.889045 (+ 613us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:09:42.889071 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:09:42.890806 (+ 1735us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:09:42.892938 (+ 2132us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:09:42.892943 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:09:42.892955 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:09:42.893212 (+ 257us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:09:42.893518 (+ 306us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:09:42.893520 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:09:42.893521 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:09:42.893652 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:09:42.894105 (+ 453us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:09:42.894111 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:09:42.894331 (+ 220us) client_negotiation.cc:769] Sending connection context
0620 06:09:42.894481 (+ 150us) client_negotiation.cc:240] Negotiation successful
0620 06:09:42.894732 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":233,"thread_start_us":98,"threads_started":1}
I20260620 06:09:42.897042 27255 heartbeater.cc:347] Connected to a master server at 127.11.102.62:41831
I20260620 06:09:42.897300 27255 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.102.2@KRBTEST.COM" created.
I20260620 06:09:42.897835 27255 heartbeater.cc:511] Master 127.11.102.62:41831 requested a full tablet report, sending...
I20260620 06:09:42.899410 27064 ts_manager.cc:194] Registered new tserver with Master: 3fd44bea82cc4e89bfc9ada83a1fd302 (127.11.102.1:37171)
I20260620 06:09:42.900542 27064 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.102.1@KRBTEST.COM'} at 127.11.102.1:41443
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.102.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935603120653-11672-0/krb5kdc/kudu_127.11.102.2.keytab.
Entry for principal kudu/127.11.102.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935603120653-11672-0/krb5kdc/kudu_127.11.102.2.keytab.
I20260620 06:09:42.912539 11672 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9HMp43/build/debug/bin/kudu
/tmp/dist-test-task9HMp43/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935603120653-11672-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935603120653-11672-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935603120653-11672-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935603120653-11672-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.102.2:0
--local_ip_for_outbound_sockets=127.11.102.2
--webserver_interface=127.11.102.2
--webserver_port=0
--tserver_master_addrs=127.11.102.62:41831
--keytab_file=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935603120653-11672-0/krb5kdc/kudu_127.11.102.2.keytab
--principal=kudu/127.11.102.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.102.20:38439
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935603120653-11672-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935603120653-11672-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935603120653-11672-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 06:09:43.021471 27260 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:09:43.021721 27260 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:09:43.021795 27260 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:09:43.025964 27260 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:09:43.026018 27260 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:09:43.026115 27260 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.102.2
I20260620 06:09:43.031215 27260 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.102.20:38439
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935603120653-11672-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935603120653-11672-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935603120653-11672-0/krb5kdc/kudu_127.11.102.2.keytab
--principal=kudu/127.11.102.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.102.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935603120653-11672-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.11.102.2
--webserver_port=0
--tserver_master_addrs=127.11.102.62:41831
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27260
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.102.2
--log_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935603120653-11672-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 05:28:48 UTC on ccycloud.quasar-fixhms.root.comops.site
build id 4457020
I20260620 06:09:43.032245 27260 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:09:43.033214 27260 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 06:09:43.043174 27268 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:09:43.043167 27265 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:09:43.043167 27266 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:09:43.043413 27260 server_base.cc:1034] running on GCE node
I20260620 06:09:43.043922 27260 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:09:43.044574 27260 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:09:43.045758 27260 hybrid_clock.cc:648] HybridClock initialized: now 1781935783045741 us; error 26 us; skew 500 ppm
Jun 20 06:09:43 dist-test-slave-9hm5 krb5kdc[27019](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935783, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.102.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 06:09:43.048950 27260 init.cc:377] Logged in from keytab as kudu/127.11.102.2@KRBTEST.COM (short username kudu)
I20260620 06:09:43.049839 27260 webserver.cc:460] Webserver started at http://127.11.102.2:42091/ using document root <none> and password file <none>
I20260620 06:09:43.050408 27260 fs_manager.cc:359] Metadata directory not provided
I20260620 06:09:43.050474 27260 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:09:43.050714 27260 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 06:09:43.052897 27260 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935603120653-11672-0/minicluster-data/ts-1/data/instance:
uuid: "cfae6937c171468db2b68d87b1149dea"
format_stamp: "Formatted at 2026-06-20 06:09:43 on dist-test-slave-9hm5"
server_key: "4e9bccdc31a644917c436f65b57a1180"
server_key_iv: "bf1fa8eb3d43a242ea056778541df575"
server_key_version: "encryptionkey@0"
I20260620 06:09:43.053480 27260 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935603120653-11672-0/minicluster-data/ts-1/wal/instance:
uuid: "cfae6937c171468db2b68d87b1149dea"
format_stamp: "Formatted at 2026-06-20 06:09:43 on dist-test-slave-9hm5"
server_key: "4e9bccdc31a644917c436f65b57a1180"
server_key_iv: "bf1fa8eb3d43a242ea056778541df575"
server_key_version: "encryptionkey@0"
I20260620 06:09:43.057384 27260 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260620 06:09:43.059901 27275 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:09:43.061177 27260 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260620 06:09:43.061312 27260 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935603120653-11672-0/minicluster-data/ts-1/data,/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935603120653-11672-0/minicluster-data/ts-1/wal
uuid: "cfae6937c171468db2b68d87b1149dea"
format_stamp: "Formatted at 2026-06-20 06:09:43 on dist-test-slave-9hm5"
server_key: "4e9bccdc31a644917c436f65b57a1180"
server_key_iv: "bf1fa8eb3d43a242ea056778541df575"
server_key_version: "encryptionkey@0"
I20260620 06:09:43.061430 27260 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935603120653-11672-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935603120653-11672-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935603120653-11672-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:09:43.083299 27260 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:09:43.086993 27260 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:09:43.087230 27260 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:09:43.087913 27260 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 06:09:43.088899 27260 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 06:09:43.088950 27260 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 06:09:43.089020 27260 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 06:09:43.089066 27260 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 06:09:43.098733 27260 rpc_server.cc:307] RPC server started. Bound to: 127.11.102.2:40107
I20260620 06:09:43.098766 27388 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.102.2:40107 every 8 connection(s)
I20260620 06:09:43.099833 27260 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935603120653-11672-0/minicluster-data/ts-1/data/info.pb
I20260620 06:09:43.108876 11672 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9HMp43/build/debug/bin/kudu as pid 27260
I20260620 06:09:43.109017 11672 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935603120653-11672-0/minicluster-data/ts-1/wal/instance
I20260620 06:09:43.109340 11672 external_mini_cluster.cc:1428] Setting key 64b1e6f61b8c6ebb5669454f9f503baa
Jun 20 06:09:43 dist-test-slave-9hm5 krb5kdc[27019](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935783, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.102.2@KRBTEST.COM for kudu/127.11.102.62@KRBTEST.COM
I20260620 06:09:43.115232 27103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:09:43.101794 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.102.2:48183 (local address 127.11.102.62:41831)
0620 06:09:43.101931 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0620 06:09:43.101934 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 06:09:43.102866 (+ 932us) server_negotiation.cc:407] Connection header received
0620 06:09:43.104187 (+ 1321us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:09:43.104190 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:09:43.104261 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:09:43.104350 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:09:43.105527 (+ 1177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:09:43.106107 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:09:43.107400 (+ 1293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:09:43.107779 (+ 379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:09:43.107874 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:09:43.111152 (+ 3278us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:09:43.111155 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:09:43.111157 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:09:43.111181 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:09:43.112882 (+ 1701us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:09:43.113397 (+ 515us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:09:43.113401 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:09:43.113403 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:09:43.113493 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:09:43.113919 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:09:43.113922 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:09:43.113923 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:09:43.114120 (+ 197us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:09:43.114207 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0620 06:09:43.114883 (+ 676us) server_negotiation.cc:299] Negotiation successful
0620 06:09:43.114998 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260620 06:09:43.116398 27391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:09:43.102237 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.102.62:41831 (local address 127.11.102.2:48183)
0620 06:09:43.102698 (+ 461us) negotiation.cc:107] Waiting for socket to connect
0620 06:09:43.102740 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0620 06:09:43.103961 (+ 1221us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:09:43.104510 (+ 549us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:09:43.104516 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:09:43.104894 (+ 378us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:09:43.105375 (+ 481us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:09:43.105387 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:09:43.106320 (+ 933us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:09:43.106324 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:09:43.107222 (+ 898us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:09:43.107231 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:09:43.107918 (+ 687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:09:43.107921 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:09:43.108154 (+ 233us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:09:43.108865 (+ 711us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:09:43.108888 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:09:43.110979 (+ 2091us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:09:43.113003 (+ 2024us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:09:43.113009 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:09:43.113033 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:09:43.113290 (+ 257us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:09:43.113587 (+ 297us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:09:43.113593 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:09:43.113598 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:09:43.113805 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:09:43.114236 (+ 431us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:09:43.114246 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:09:43.114585 (+ 339us) client_negotiation.cc:769] Sending connection context
0620 06:09:43.114822 (+ 237us) client_negotiation.cc:240] Negotiation successful
0620 06:09:43.115208 (+ 386us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":260,"thread_start_us":109,"threads_started":1}
I20260620 06:09:43.117645 27389 heartbeater.cc:347] Connected to a master server at 127.11.102.62:41831
I20260620 06:09:43.117941 27389 heartbeater.cc:464] Registering TS with master...
I20260620 06:09:43.118541 27389 heartbeater.cc:511] Master 127.11.102.62:41831 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.11.102.3@KRBTEST.COM; defaulting to no policy
I20260620 06:09:43.119757 27064 ts_manager.cc:194] Registered new tserver with Master: cfae6937c171468db2b68d87b1149dea (127.11.102.2:40107)
I20260620 06:09:43.120330 27064 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.102.2@KRBTEST.COM'} at 127.11.102.2:48183
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.102.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.102.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935603120653-11672-0/krb5kdc/kudu_127.11.102.3.keytab.
Entry for principal kudu/127.11.102.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935603120653-11672-0/krb5kdc/kudu_127.11.102.3.keytab.
I20260620 06:09:43.138538 11672 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9HMp43/build/debug/bin/kudu
/tmp/dist-test-task9HMp43/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935603120653-11672-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935603120653-11672-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935603120653-11672-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935603120653-11672-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.102.3:0
--local_ip_for_outbound_sockets=127.11.102.3
--webserver_interface=127.11.102.3
--webserver_port=0
--tserver_master_addrs=127.11.102.62:41831
--keytab_file=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935603120653-11672-0/krb5kdc/kudu_127.11.102.3.keytab
--principal=kudu/127.11.102.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.102.20:38439
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935603120653-11672-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935603120653-11672-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935603120653-11672-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 06:09:43.248786 27394 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:09:43.249065 27394 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:09:43.249177 27394 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:09:43.253433 27394 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:09:43.253517 27394 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:09:43.253625 27394 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.102.3
I20260620 06:09:43.258908 27394 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.102.20:38439
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935603120653-11672-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935603120653-11672-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935603120653-11672-0/krb5kdc/kudu_127.11.102.3.keytab
--principal=kudu/127.11.102.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.102.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935603120653-11672-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.11.102.3
--webserver_port=0
--tserver_master_addrs=127.11.102.62:41831
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27394
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.102.3
--log_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935603120653-11672-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 05:28:48 UTC on ccycloud.quasar-fixhms.root.comops.site
build id 4457020
I20260620 06:09:43.260072 27394 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:09:43.261150 27394 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 06:09:43.270991 27400 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:09:43.271027 27394 server_base.cc:1034] running on GCE node
W20260620 06:09:43.270951 27399 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:09:43.270958 27402 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:09:43.271605 27394 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:09:43.272279 27394 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:09:43.273470 27394 hybrid_clock.cc:648] HybridClock initialized: now 1781935783273449 us; error 37 us; skew 500 ppm
Jun 20 06:09:43 dist-test-slave-9hm5 krb5kdc[27019](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935783, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.102.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 06:09:43.276273 27394 init.cc:377] Logged in from keytab as kudu/127.11.102.3@KRBTEST.COM (short username kudu)
I20260620 06:09:43.277089 27394 webserver.cc:460] Webserver started at http://127.11.102.3:42325/ using document root <none> and password file <none>
I20260620 06:09:43.277647 27394 fs_manager.cc:359] Metadata directory not provided
I20260620 06:09:43.277740 27394 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:09:43.277980 27394 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 06:09:43.280058 27394 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935603120653-11672-0/minicluster-data/ts-2/data/instance:
uuid: "5600989c2f634c4cb630839afa98c1a3"
format_stamp: "Formatted at 2026-06-20 06:09:43 on dist-test-slave-9hm5"
server_key: "7c752eff990c445a6e4932608495d08e"
server_key_iv: "0b3b1b24d5500c0211dfa70a90f90c34"
server_key_version: "encryptionkey@0"
I20260620 06:09:43.280607 27394 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935603120653-11672-0/minicluster-data/ts-2/wal/instance:
uuid: "5600989c2f634c4cb630839afa98c1a3"
format_stamp: "Formatted at 2026-06-20 06:09:43 on dist-test-slave-9hm5"
server_key: "7c752eff990c445a6e4932608495d08e"
server_key_iv: "0b3b1b24d5500c0211dfa70a90f90c34"
server_key_version: "encryptionkey@0"
I20260620 06:09:43.284192 27394 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260620 06:09:43.286509 27409 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:09:43.287853 27394 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 06:09:43.287999 27394 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935603120653-11672-0/minicluster-data/ts-2/data,/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935603120653-11672-0/minicluster-data/ts-2/wal
uuid: "5600989c2f634c4cb630839afa98c1a3"
format_stamp: "Formatted at 2026-06-20 06:09:43 on dist-test-slave-9hm5"
server_key: "7c752eff990c445a6e4932608495d08e"
server_key_iv: "0b3b1b24d5500c0211dfa70a90f90c34"
server_key_version: "encryptionkey@0"
I20260620 06:09:43.288112 27394 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935603120653-11672-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935603120653-11672-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935603120653-11672-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:09:43.300649 27394 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:09:43.303496 27394 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:09:43.303735 27394 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:09:43.304282 27394 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 06:09:43.305063 27394 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 06:09:43.305132 27394 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 06:09:43.305195 27394 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 06:09:43.305250 27394 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 06:09:43.314131 27394 rpc_server.cc:307] RPC server started. Bound to: 127.11.102.3:43283
I20260620 06:09:43.314175 27522 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.102.3:43283 every 8 connection(s)
I20260620 06:09:43.314996 27394 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935603120653-11672-0/minicluster-data/ts-2/data/info.pb
Jun 20 06:09:43 dist-test-slave-9hm5 krb5kdc[27019](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935783, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.102.3@KRBTEST.COM for kudu/127.11.102.62@KRBTEST.COM
I20260620 06:09:43.324862 11672 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9HMp43/build/debug/bin/kudu as pid 27394
I20260620 06:09:43.325004 11672 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935603120653-11672-0/minicluster-data/ts-2/wal/instance
I20260620 06:09:43.325309 11672 external_mini_cluster.cc:1428] Setting key 565f04d5b3266e704463184aaebffaa4
I20260620 06:09:43.328774 27103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:09:43.316710 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.102.3:39415 (local address 127.11.102.62:41831)
0620 06:09:43.316848 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0620 06:09:43.316852 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 06:09:43.317657 (+ 805us) server_negotiation.cc:407] Connection header received
0620 06:09:43.318853 (+ 1196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:09:43.318855 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:09:43.318903 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:09:43.318979 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:09:43.320180 (+ 1201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:09:43.320715 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:09:43.321849 (+ 1134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:09:43.322057 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:09:43.322138 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:09:43.324652 (+ 2514us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:09:43.324655 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:09:43.324657 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:09:43.324684 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:09:43.326496 (+ 1812us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:09:43.327236 (+ 740us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:09:43.327241 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:09:43.327257 (+ 16us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:09:43.327352 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:09:43.327785 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:09:43.327788 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:09:43.327789 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:09:43.327980 (+ 191us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:09:43.328070 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0620 06:09:43.328560 (+ 490us) server_negotiation.cc:299] Negotiation successful
0620 06:09:43.328661 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260620 06:09:43.329612 27525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:09:43.317101 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.102.62:41831 (local address 127.11.102.3:39415)
0620 06:09:43.317507 (+ 406us) negotiation.cc:107] Waiting for socket to connect
0620 06:09:43.317546 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0620 06:09:43.318659 (+ 1113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:09:43.319165 (+ 506us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:09:43.319171 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:09:43.319517 (+ 346us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:09:43.320046 (+ 529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:09:43.320059 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:09:43.320847 (+ 788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:09:43.320850 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:09:43.321734 (+ 884us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:09:43.321743 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:09:43.322206 (+ 463us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:09:43.322209 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:09:43.322352 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:09:43.322970 (+ 618us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:09:43.322986 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:09:43.324456 (+ 1470us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:09:43.326648 (+ 2192us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:09:43.326653 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:09:43.326665 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:09:43.326995 (+ 330us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:09:43.327479 (+ 484us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:09:43.327482 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:09:43.327483 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:09:43.327688 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:09:43.328088 (+ 400us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:09:43.328093 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:09:43.328338 (+ 245us) client_negotiation.cc:769] Sending connection context
0620 06:09:43.328489 (+ 151us) client_negotiation.cc:240] Negotiation successful
0620 06:09:43.328746 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":236,"thread_start_us":98,"threads_started":1}
I20260620 06:09:43.330654 27523 heartbeater.cc:347] Connected to a master server at 127.11.102.62:41831
I20260620 06:09:43.330897 27523 heartbeater.cc:464] Registering TS with master...
I20260620 06:09:43.331385 27523 heartbeater.cc:511] Master 127.11.102.62:41831 requested a full tablet report, sending...
I20260620 06:09:43.332386 27064 ts_manager.cc:194] Registered new tserver with Master: 5600989c2f634c4cb630839afa98c1a3 (127.11.102.3:43283)
I20260620 06:09:43.332935 27064 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.102.3@KRBTEST.COM'} at 127.11.102.3:39415
I20260620 06:09:43.339156 11672 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 06:09:43.347756 27103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:09:43.341057 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33372 (local address 127.11.102.62:41831)
0620 06:09:43.341215 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0620 06:09:43.341221 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 06:09:43.341339 (+ 118us) server_negotiation.cc:407] Connection header received
0620 06:09:43.341436 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:09:43.341438 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:09:43.341472 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:09:43.341535 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:09:43.342158 (+ 623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:09:43.342647 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:09:43.343410 (+ 763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:09:43.343701 (+ 291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:09:43.343776 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:09:43.345006 (+ 1230us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:09:43.345009 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:09:43.345012 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:09:43.345033 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:09:43.346274 (+ 1241us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:09:43.346639 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:09:43.346641 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:09:43.346642 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:09:43.346710 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:09:43.346936 (+ 226us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:09:43.346938 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:09:43.346939 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:09:43.347147 (+ 208us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:09:43.347247 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0620 06:09:43.347417 (+ 170us) server_negotiation.cc:299] Negotiation successful
0620 06:09:43.347503 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260620 06:09:43.350461 27064 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33372:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 06:09:43.353613 27064 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 06:09:43.368034 27537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:09:43.362196 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44954 (local address 127.11.102.1:37171)
0620 06:09:43.362535 (+ 339us) server_negotiation.cc:206] Beginning negotiation
0620 06:09:43.362552 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0620 06:09:43.363412 (+ 860us) server_negotiation.cc:407] Connection header received
0620 06:09:43.363644 (+ 232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:09:43.363654 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:09:43.363873 (+ 219us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:09:43.364039 (+ 166us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 06:09:43.364337 (+ 298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:09:43.365313 (+ 976us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:09:43.366787 (+ 1474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:09:43.367333 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:09:43.367451 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:09:43.367594 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0620 06:09:43.367791 (+ 197us) server_negotiation.cc:299] Negotiation successful
0620 06:09:43.367884 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":242,"thread_start_us":79,"threads_started":1}
I20260620 06:09:43.368023 27535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:09:43.362020 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.102.1:37171 (local address 127.0.0.1:44954)
0620 06:09:43.363295 (+ 1275us) negotiation.cc:107] Waiting for socket to connect
0620 06:09:43.363310 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 06:09:43.363454 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:09:43.364036 (+ 582us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:09:43.364042 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:09:43.364070 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 06:09:43.364217 (+ 147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:09:43.364225 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:09:43.365461 (+ 1236us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:09:43.365465 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:09:43.366662 (+ 1197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:09:43.366672 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:09:43.367462 (+ 790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:09:43.367465 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:09:43.367566 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:09:43.367601 (+ 35us) client_negotiation.cc:769] Sending connection context
0620 06:09:43.367757 (+ 156us) client_negotiation.cc:240] Negotiation successful
0620 06:09:43.367876 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1216,"thread_start_us":86,"threads_started":1}
I20260620 06:09:43.369153 27536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:09:43.362001 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55586 (local address 127.11.102.3:43283)
0620 06:09:43.362767 (+ 766us) server_negotiation.cc:206] Beginning negotiation
0620 06:09:43.362775 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 06:09:43.363159 (+ 384us) server_negotiation.cc:407] Connection header received
0620 06:09:43.363819 (+ 660us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:09:43.363824 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:09:43.364003 (+ 179us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:09:43.364127 (+ 124us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 06:09:43.364568 (+ 441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:09:43.365714 (+ 1146us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:09:43.367468 (+ 1754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:09:43.368329 (+ 861us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:09:43.368410 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:09:43.368562 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0620 06:09:43.368908 (+ 346us) server_negotiation.cc:299] Negotiation successful
0620 06:09:43.369002 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":684,"thread_start_us":156,"threads_started":1}
I20260620 06:09:43.369499 27534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:09:43.361880 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.102.3:43283 (local address 127.0.0.1:55586)
0620 06:09:43.363015 (+ 1135us) negotiation.cc:107] Waiting for socket to connect
0620 06:09:43.363063 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0620 06:09:43.363661 (+ 598us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:09:43.364309 (+ 648us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:09:43.364314 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:09:43.364333 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 06:09:43.364412 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:09:43.364416 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:09:43.365871 (+ 1455us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:09:43.365873 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:09:43.367326 (+ 1453us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:09:43.367337 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:09:43.368672 (+ 1335us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:09:43.368675 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:09:43.368761 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:09:43.368781 (+ 20us) client_negotiation.cc:769] Sending connection context
0620 06:09:43.369299 (+ 518us) client_negotiation.cc:240] Negotiation successful
0620 06:09:43.369377 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":970,"mutex_wait_us":100,"thread_start_us":85,"threads_started":1}
I20260620 06:09:43.371160 27189 tablet_service.cc:1467] Processing CreateTablet for tablet 11aae1fd98d748b89baa742e77198219 (DEFAULT_TABLE table=test-table [id=e26f28e3aa7045758855c1c72065084c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 06:09:43.372160 27189 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 11aae1fd98d748b89baa742e77198219. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 06:09:43.371887 27457 tablet_service.cc:1467] Processing CreateTablet for tablet 11aae1fd98d748b89baa742e77198219 (DEFAULT_TABLE table=test-table [id=e26f28e3aa7045758855c1c72065084c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 06:09:43.372864 27457 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 11aae1fd98d748b89baa742e77198219. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 06:09:43.378239 27538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:09:43.362865 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.102.2:40107 (local address 127.0.0.1:41630)
0620 06:09:43.363574 (+ 709us) negotiation.cc:107] Waiting for socket to connect
0620 06:09:43.363587 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 06:09:43.363689 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:09:43.364224 (+ 535us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:09:43.364227 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:09:43.364244 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 06:09:43.364372 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:09:43.364382 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:09:43.365869 (+ 1487us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:09:43.365872 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:09:43.367326 (+ 1454us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:09:43.367337 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:09:43.377801 (+ 10464us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:09:43.377805 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:09:43.377901 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:09:43.377927 (+ 26us) client_negotiation.cc:769] Sending connection context
0620 06:09:43.378011 (+ 84us) client_negotiation.cc:240] Negotiation successful
0620 06:09:43.378099 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":650,"mutex_wait_us":183,"thread_start_us":97,"threads_started":1}
I20260620 06:09:43.378319 27540 tablet_bootstrap.cc:492] T 11aae1fd98d748b89baa742e77198219 P 3fd44bea82cc4e89bfc9ada83a1fd302: Bootstrap starting.
I20260620 06:09:43.378794 27539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:09:43.363037 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41630 (local address 127.11.102.2:40107)
0620 06:09:43.363445 (+ 408us) server_negotiation.cc:206] Beginning negotiation
0620 06:09:43.363456 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 06:09:43.363692 (+ 236us) server_negotiation.cc:407] Connection header received
0620 06:09:43.363795 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:09:43.363800 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:09:43.363952 (+ 152us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:09:43.364076 (+ 124us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 06:09:43.364505 (+ 429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:09:43.365725 (+ 1220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:09:43.367468 (+ 1743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:09:43.377436 (+ 9968us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:09:43.377556 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:09:43.377705 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0620 06:09:43.378516 (+ 811us) server_negotiation.cc:299] Negotiation successful
0620 06:09:43.378632 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":321,"thread_start_us":111,"threads_started":1}
I20260620 06:09:43.378942 27541 tablet_bootstrap.cc:492] T 11aae1fd98d748b89baa742e77198219 P 5600989c2f634c4cb630839afa98c1a3: Bootstrap starting.
I20260620 06:09:43.381278 27540 tablet_bootstrap.cc:654] T 11aae1fd98d748b89baa742e77198219 P 3fd44bea82cc4e89bfc9ada83a1fd302: Neither blocks nor log segments found. Creating new log.
I20260620 06:09:43.381510 27541 tablet_bootstrap.cc:654] T 11aae1fd98d748b89baa742e77198219 P 5600989c2f634c4cb630839afa98c1a3: Neither blocks nor log segments found. Creating new log.
I20260620 06:09:43.382334 27540 log.cc:824] T 11aae1fd98d748b89baa742e77198219 P 3fd44bea82cc4e89bfc9ada83a1fd302: Log is configured to *not* fsync() on all Append() calls
I20260620 06:09:43.382380 27541 log.cc:824] T 11aae1fd98d748b89baa742e77198219 P 5600989c2f634c4cb630839afa98c1a3: Log is configured to *not* fsync() on all Append() calls
I20260620 06:09:43.382243 27323 tablet_service.cc:1467] Processing CreateTablet for tablet 11aae1fd98d748b89baa742e77198219 (DEFAULT_TABLE table=test-table [id=e26f28e3aa7045758855c1c72065084c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 06:09:43.383340 27323 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 11aae1fd98d748b89baa742e77198219. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 06:09:43.384405 27540 tablet_bootstrap.cc:492] T 11aae1fd98d748b89baa742e77198219 P 3fd44bea82cc4e89bfc9ada83a1fd302: No bootstrap required, opened a new log
I20260620 06:09:43.384519 27541 tablet_bootstrap.cc:492] T 11aae1fd98d748b89baa742e77198219 P 5600989c2f634c4cb630839afa98c1a3: No bootstrap required, opened a new log
I20260620 06:09:43.384613 27540 ts_tablet_manager.cc:1397] T 11aae1fd98d748b89baa742e77198219 P 3fd44bea82cc4e89bfc9ada83a1fd302: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260620 06:09:43.384706 27541 ts_tablet_manager.cc:1397] T 11aae1fd98d748b89baa742e77198219 P 5600989c2f634c4cb630839afa98c1a3: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260620 06:09:43.387310 27540 raft_consensus.cc:348] T 11aae1fd98d748b89baa742e77198219 P 3fd44bea82cc4e89bfc9ada83a1fd302 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5600989c2f634c4cb630839afa98c1a3" member_type: VOTER last_known_addr { host: "127.11.102.3" port: 43283 } } peers { permanent_uuid: "3fd44bea82cc4e89bfc9ada83a1fd302" member_type: VOTER last_known_addr { host: "127.11.102.1" port: 37171 } } peers { permanent_uuid: "cfae6937c171468db2b68d87b1149dea" member_type: VOTER last_known_addr { host: "127.11.102.2" port: 40107 } }
I20260620 06:09:43.387497 27540 raft_consensus.cc:374] T 11aae1fd98d748b89baa742e77198219 P 3fd44bea82cc4e89bfc9ada83a1fd302 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 06:09:43.387527 27540 raft_consensus.cc:729] T 11aae1fd98d748b89baa742e77198219 P 3fd44bea82cc4e89bfc9ada83a1fd302 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3fd44bea82cc4e89bfc9ada83a1fd302, State: Initialized, Role: FOLLOWER
I20260620 06:09:43.387984 27540 consensus_queue.cc:260] T 11aae1fd98d748b89baa742e77198219 P 3fd44bea82cc4e89bfc9ada83a1fd302 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5600989c2f634c4cb630839afa98c1a3" member_type: VOTER last_known_addr { host: "127.11.102.3" port: 43283 } } peers { permanent_uuid: "3fd44bea82cc4e89bfc9ada83a1fd302" member_type: VOTER last_known_addr { host: "127.11.102.1" port: 37171 } } peers { permanent_uuid: "cfae6937c171468db2b68d87b1149dea" member_type: VOTER last_known_addr { host: "127.11.102.2" port: 40107 } }
I20260620 06:09:43.388660 27255 heartbeater.cc:503] Master 127.11.102.62:41831 was elected leader, sending a full tablet report...
I20260620 06:09:43.388940 27540 ts_tablet_manager.cc:1428] T 11aae1fd98d748b89baa742e77198219 P 3fd44bea82cc4e89bfc9ada83a1fd302: Time spent starting tablet: real 0.004s user 0.005s sys 0.000s
I20260620 06:09:43.388451 27541 raft_consensus.cc:348] T 11aae1fd98d748b89baa742e77198219 P 5600989c2f634c4cb630839afa98c1a3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5600989c2f634c4cb630839afa98c1a3" member_type: VOTER last_known_addr { host: "127.11.102.3" port: 43283 } } peers { permanent_uuid: "3fd44bea82cc4e89bfc9ada83a1fd302" member_type: VOTER last_known_addr { host: "127.11.102.1" port: 37171 } } peers { permanent_uuid: "cfae6937c171468db2b68d87b1149dea" member_type: VOTER last_known_addr { host: "127.11.102.2" port: 40107 } }
I20260620 06:09:43.389302 27541 raft_consensus.cc:374] T 11aae1fd98d748b89baa742e77198219 P 5600989c2f634c4cb630839afa98c1a3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 06:09:43.389365 27541 raft_consensus.cc:729] T 11aae1fd98d748b89baa742e77198219 P 5600989c2f634c4cb630839afa98c1a3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5600989c2f634c4cb630839afa98c1a3, State: Initialized, Role: FOLLOWER
I20260620 06:09:43.389803 27541 consensus_queue.cc:260] T 11aae1fd98d748b89baa742e77198219 P 5600989c2f634c4cb630839afa98c1a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5600989c2f634c4cb630839afa98c1a3" member_type: VOTER last_known_addr { host: "127.11.102.3" port: 43283 } } peers { permanent_uuid: "3fd44bea82cc4e89bfc9ada83a1fd302" member_type: VOTER last_known_addr { host: "127.11.102.1" port: 37171 } } peers { permanent_uuid: "cfae6937c171468db2b68d87b1149dea" member_type: VOTER last_known_addr { host: "127.11.102.2" port: 40107 } }
I20260620 06:09:43.390479 27523 heartbeater.cc:503] Master 127.11.102.62:41831 was elected leader, sending a full tablet report...
I20260620 06:09:43.390767 27541 ts_tablet_manager.cc:1428] T 11aae1fd98d748b89baa742e77198219 P 5600989c2f634c4cb630839afa98c1a3: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260620 06:09:43.391413 27546 tablet_bootstrap.cc:492] T 11aae1fd98d748b89baa742e77198219 P cfae6937c171468db2b68d87b1149dea: Bootstrap starting.
I20260620 06:09:43.394024 27546 tablet_bootstrap.cc:654] T 11aae1fd98d748b89baa742e77198219 P cfae6937c171468db2b68d87b1149dea: Neither blocks nor log segments found. Creating new log.
I20260620 06:09:43.394763 27546 log.cc:824] T 11aae1fd98d748b89baa742e77198219 P cfae6937c171468db2b68d87b1149dea: Log is configured to *not* fsync() on all Append() calls
I20260620 06:09:43.396462 27546 tablet_bootstrap.cc:492] T 11aae1fd98d748b89baa742e77198219 P cfae6937c171468db2b68d87b1149dea: No bootstrap required, opened a new log
I20260620 06:09:43.396652 27546 ts_tablet_manager.cc:1397] T 11aae1fd98d748b89baa742e77198219 P cfae6937c171468db2b68d87b1149dea: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260620 06:09:43.399538 27546 raft_consensus.cc:348] T 11aae1fd98d748b89baa742e77198219 P cfae6937c171468db2b68d87b1149dea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5600989c2f634c4cb630839afa98c1a3" member_type: VOTER last_known_addr { host: "127.11.102.3" port: 43283 } } peers { permanent_uuid: "3fd44bea82cc4e89bfc9ada83a1fd302" member_type: VOTER last_known_addr { host: "127.11.102.1" port: 37171 } } peers { permanent_uuid: "cfae6937c171468db2b68d87b1149dea" member_type: VOTER last_known_addr { host: "127.11.102.2" port: 40107 } }
I20260620 06:09:43.399813 27546 raft_consensus.cc:374] T 11aae1fd98d748b89baa742e77198219 P cfae6937c171468db2b68d87b1149dea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 06:09:43.399896 27546 raft_consensus.cc:729] T 11aae1fd98d748b89baa742e77198219 P cfae6937c171468db2b68d87b1149dea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cfae6937c171468db2b68d87b1149dea, State: Initialized, Role: FOLLOWER
I20260620 06:09:43.400286 27546 consensus_queue.cc:260] T 11aae1fd98d748b89baa742e77198219 P cfae6937c171468db2b68d87b1149dea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5600989c2f634c4cb630839afa98c1a3" member_type: VOTER last_known_addr { host: "127.11.102.3" port: 43283 } } peers { permanent_uuid: "3fd44bea82cc4e89bfc9ada83a1fd302" member_type: VOTER last_known_addr { host: "127.11.102.1" port: 37171 } } peers { permanent_uuid: "cfae6937c171468db2b68d87b1149dea" member_type: VOTER last_known_addr { host: "127.11.102.2" port: 40107 } }
I20260620 06:09:43.400864 27389 heartbeater.cc:503] Master 127.11.102.62:41831 was elected leader, sending a full tablet report...
I20260620 06:09:43.401087 27546 ts_tablet_manager.cc:1428] T 11aae1fd98d748b89baa742e77198219 P cfae6937c171468db2b68d87b1149dea: Time spent starting tablet: real 0.004s user 0.005s sys 0.000s
W20260620 06:09:43.566160 27524 tablet.cc:2369] T 11aae1fd98d748b89baa742e77198219 P 5600989c2f634c4cb630839afa98c1a3: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 06:09:43.601019 27390 tablet.cc:2369] T 11aae1fd98d748b89baa742e77198219 P cfae6937c171468db2b68d87b1149dea: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 06:09:43.619040 27548 raft_consensus.cc:482] T 11aae1fd98d748b89baa742e77198219 P cfae6937c171468db2b68d87b1149dea [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 06:09:43.619287 27548 raft_consensus.cc:504] T 11aae1fd98d748b89baa742e77198219 P cfae6937c171468db2b68d87b1149dea [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5600989c2f634c4cb630839afa98c1a3" member_type: VOTER last_known_addr { host: "127.11.102.3" port: 43283 } } peers { permanent_uuid: "3fd44bea82cc4e89bfc9ada83a1fd302" member_type: VOTER last_known_addr { host: "127.11.102.1" port: 37171 } } peers { permanent_uuid: "cfae6937c171468db2b68d87b1149dea" member_type: VOTER last_known_addr { host: "127.11.102.2" port: 40107 } }
I20260620 06:09:43.620553 27548 leader_election.cc:288] T 11aae1fd98d748b89baa742e77198219 P cfae6937c171468db2b68d87b1149dea [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5600989c2f634c4cb630839afa98c1a3 (127.11.102.3:43283), 3fd44bea82cc4e89bfc9ada83a1fd302 (127.11.102.1:37171)
I20260620 06:09:43.624403 27536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:09:43.620911 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.102.2:46175 (local address 127.11.102.3:43283)
0620 06:09:43.621051 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0620 06:09:43.621056 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 06:09:43.621228 (+ 172us) server_negotiation.cc:407] Connection header received
0620 06:09:43.621395 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:09:43.621403 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:09:43.621443 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:09:43.621527 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 06:09:43.621792 (+ 265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:09:43.622297 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:09:43.623349 (+ 1052us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:09:43.623798 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:09:43.623924 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:09:43.623982 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0620 06:09:43.624183 (+ 201us) server_negotiation.cc:299] Negotiation successful
0620 06:09:43.624258 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260620 06:09:43.624410 27537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:09:43.620984 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.102.2:57395 (local address 127.11.102.1:37171)
0620 06:09:43.621110 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0620 06:09:43.621114 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 06:09:43.621299 (+ 185us) server_negotiation.cc:407] Connection header received
0620 06:09:43.621458 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:09:43.621466 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:09:43.621510 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:09:43.621597 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 06:09:43.621881 (+ 284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:09:43.622366 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:09:43.623386 (+ 1020us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:09:43.623800 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:09:43.623928 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:09:43.623989 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0620 06:09:43.624209 (+ 220us) server_negotiation.cc:299] Negotiation successful
0620 06:09:43.624282 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260620 06:09:43.624434 27550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:09:43.620889 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.102.1:37171 (local address 127.11.102.2:57395)
0620 06:09:43.621177 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0620 06:09:43.621193 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 06:09:43.621330 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:09:43.621605 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:09:43.621608 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:09:43.621631 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 06:09:43.621765 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:09:43.621770 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:09:43.622479 (+ 709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:09:43.622484 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:09:43.623256 (+ 772us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:09:43.623264 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:09:43.623920 (+ 656us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:09:43.623924 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:09:43.624087 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:09:43.624104 (+ 17us) client_negotiation.cc:769] Sending connection context
0620 06:09:43.624193 (+ 89us) client_negotiation.cc:240] Negotiation successful
0620 06:09:43.624280 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":226,"mutex_wait_us":46,"thread_start_us":79,"threads_started":1}
I20260620 06:09:43.624447 27549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:09:43.620760 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.102.3:43283 (local address 127.11.102.2:46175)
0620 06:09:43.621116 (+ 356us) negotiation.cc:107] Waiting for socket to connect
0620 06:09:43.621132 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 06:09:43.621285 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:09:43.621539 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:09:43.621541 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:09:43.621556 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 06:09:43.621678 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:09:43.621683 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:09:43.622412 (+ 729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:09:43.622415 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:09:43.623226 (+ 811us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:09:43.623235 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:09:43.623919 (+ 684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:09:43.623923 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:09:43.624052 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:09:43.624078 (+ 26us) client_negotiation.cc:769] Sending connection context
0620 06:09:43.624167 (+ 89us) client_negotiation.cc:240] Negotiation successful
0620 06:09:43.624272 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":281,"mutex_wait_us":26,"thread_start_us":104,"threads_started":1}
I20260620 06:09:43.625128 27209 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11aae1fd98d748b89baa742e77198219" candidate_uuid: "cfae6937c171468db2b68d87b1149dea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3fd44bea82cc4e89bfc9ada83a1fd302" is_pre_election: true
I20260620 06:09:43.625166 27477 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11aae1fd98d748b89baa742e77198219" candidate_uuid: "cfae6937c171468db2b68d87b1149dea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5600989c2f634c4cb630839afa98c1a3" is_pre_election: true
I20260620 06:09:43.625461 27477 raft_consensus.cc:2454] T 11aae1fd98d748b89baa742e77198219 P 5600989c2f634c4cb630839afa98c1a3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cfae6937c171468db2b68d87b1149dea in term 0.
I20260620 06:09:43.625463 27209 raft_consensus.cc:2454] T 11aae1fd98d748b89baa742e77198219 P 3fd44bea82cc4e89bfc9ada83a1fd302 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cfae6937c171468db2b68d87b1149dea in term 0.
I20260620 06:09:43.625949 27277 leader_election.cc:302] T 11aae1fd98d748b89baa742e77198219 P cfae6937c171468db2b68d87b1149dea [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 3fd44bea82cc4e89bfc9ada83a1fd302, 5600989c2f634c4cb630839afa98c1a3, cfae6937c171468db2b68d87b1149dea; no voters:
I20260620 06:09:43.626209 27548 raft_consensus.cc:2781] T 11aae1fd98d748b89baa742e77198219 P cfae6937c171468db2b68d87b1149dea [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 06:09:43.626305 27548 raft_consensus.cc:482] T 11aae1fd98d748b89baa742e77198219 P cfae6937c171468db2b68d87b1149dea [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 06:09:43.626366 27548 raft_consensus.cc:3037] T 11aae1fd98d748b89baa742e77198219 P cfae6937c171468db2b68d87b1149dea [term 0 FOLLOWER]: Advancing to term 1
I20260620 06:09:43.627382 27548 raft_consensus.cc:504] T 11aae1fd98d748b89baa742e77198219 P cfae6937c171468db2b68d87b1149dea [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5600989c2f634c4cb630839afa98c1a3" member_type: VOTER last_known_addr { host: "127.11.102.3" port: 43283 } } peers { permanent_uuid: "3fd44bea82cc4e89bfc9ada83a1fd302" member_type: VOTER last_known_addr { host: "127.11.102.1" port: 37171 } } peers { permanent_uuid: "cfae6937c171468db2b68d87b1149dea" member_type: VOTER last_known_addr { host: "127.11.102.2" port: 40107 } }
I20260620 06:09:43.627770 27548 leader_election.cc:288] T 11aae1fd98d748b89baa742e77198219 P cfae6937c171468db2b68d87b1149dea [CANDIDATE]: Term 1 election: Requested vote from peers 5600989c2f634c4cb630839afa98c1a3 (127.11.102.3:43283), 3fd44bea82cc4e89bfc9ada83a1fd302 (127.11.102.1:37171)
I20260620 06:09:43.628191 27477 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11aae1fd98d748b89baa742e77198219" candidate_uuid: "cfae6937c171468db2b68d87b1149dea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5600989c2f634c4cb630839afa98c1a3"
I20260620 06:09:43.628202 27209 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11aae1fd98d748b89baa742e77198219" candidate_uuid: "cfae6937c171468db2b68d87b1149dea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3fd44bea82cc4e89bfc9ada83a1fd302"
I20260620 06:09:43.628300 27477 raft_consensus.cc:3037] T 11aae1fd98d748b89baa742e77198219 P 5600989c2f634c4cb630839afa98c1a3 [term 0 FOLLOWER]: Advancing to term 1
I20260620 06:09:43.628338 27209 raft_consensus.cc:3037] T 11aae1fd98d748b89baa742e77198219 P 3fd44bea82cc4e89bfc9ada83a1fd302 [term 0 FOLLOWER]: Advancing to term 1
I20260620 06:09:43.629333 27477 raft_consensus.cc:2454] T 11aae1fd98d748b89baa742e77198219 P 5600989c2f634c4cb630839afa98c1a3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cfae6937c171468db2b68d87b1149dea in term 1.
I20260620 06:09:43.629427 27209 raft_consensus.cc:2454] T 11aae1fd98d748b89baa742e77198219 P 3fd44bea82cc4e89bfc9ada83a1fd302 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cfae6937c171468db2b68d87b1149dea in term 1.
I20260620 06:09:43.629631 27277 leader_election.cc:302] T 11aae1fd98d748b89baa742e77198219 P cfae6937c171468db2b68d87b1149dea [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5600989c2f634c4cb630839afa98c1a3, cfae6937c171468db2b68d87b1149dea; no voters:
I20260620 06:09:43.629845 27548 raft_consensus.cc:2781] T 11aae1fd98d748b89baa742e77198219 P cfae6937c171468db2b68d87b1149dea [term 1 FOLLOWER]: Leader election won for term 1
I20260620 06:09:43.630018 27548 raft_consensus.cc:686] T 11aae1fd98d748b89baa742e77198219 P cfae6937c171468db2b68d87b1149dea [term 1 LEADER]: Becoming Leader. State: Replica: cfae6937c171468db2b68d87b1149dea, State: Running, Role: LEADER
I20260620 06:09:43.630373 27548 consensus_queue.cc:237] T 11aae1fd98d748b89baa742e77198219 P cfae6937c171468db2b68d87b1149dea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5600989c2f634c4cb630839afa98c1a3" member_type: VOTER last_known_addr { host: "127.11.102.3" port: 43283 } } peers { permanent_uuid: "3fd44bea82cc4e89bfc9ada83a1fd302" member_type: VOTER last_known_addr { host: "127.11.102.1" port: 37171 } } peers { permanent_uuid: "cfae6937c171468db2b68d87b1149dea" member_type: VOTER last_known_addr { host: "127.11.102.2" port: 40107 } }
W20260620 06:09:43.631827 27256 tablet.cc:2369] T 11aae1fd98d748b89baa742e77198219 P 3fd44bea82cc4e89bfc9ada83a1fd302: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 06:09:43.633682 27063 catalog_manager.cc:5495] T 11aae1fd98d748b89baa742e77198219 P cfae6937c171468db2b68d87b1149dea reported cstate change: term changed from 0 to 1, leader changed from <none> to cfae6937c171468db2b68d87b1149dea (127.11.102.2). New cstate: current_term: 1 leader_uuid: "cfae6937c171468db2b68d87b1149dea" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5600989c2f634c4cb630839afa98c1a3" member_type: VOTER last_known_addr { host: "127.11.102.3" port: 43283 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3fd44bea82cc4e89bfc9ada83a1fd302" member_type: VOTER last_known_addr { host: "127.11.102.1" port: 37171 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cfae6937c171468db2b68d87b1149dea" member_type: VOTER last_known_addr { host: "127.11.102.2" port: 40107 } health_report { overall_health: HEALTHY } } }
I20260620 06:09:43.659039 27539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:09:43.655869 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41646 (local address 127.11.102.2:40107)
0620 06:09:43.656017 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0620 06:09:43.656024 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 06:09:43.656065 (+ 41us) server_negotiation.cc:407] Connection header received
0620 06:09:43.656143 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:09:43.656146 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:09:43.656224 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:09:43.656314 (+ 90us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 06:09:43.656513 (+ 199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:09:43.657002 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:09:43.657805 (+ 803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:09:43.658011 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:09:43.658088 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:09:43.658337 (+ 249us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 06:09:43.658740 (+ 403us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 06:09:43.658830 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0620 06:09:43.658881 (+ 51us) server_negotiation.cc:299] Negotiation successful
0620 06:09:43.658927 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260620 06:09:43.665145 27477 raft_consensus.cc:1261] T 11aae1fd98d748b89baa742e77198219 P 5600989c2f634c4cb630839afa98c1a3 [term 1 FOLLOWER]: Refusing update from remote peer cfae6937c171468db2b68d87b1149dea: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 06:09:43.665120 27209 raft_consensus.cc:1261] T 11aae1fd98d748b89baa742e77198219 P 3fd44bea82cc4e89bfc9ada83a1fd302 [term 1 FOLLOWER]: Refusing update from remote peer cfae6937c171468db2b68d87b1149dea: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 06:09:43.665858 27548 consensus_queue.cc:1035] T 11aae1fd98d748b89baa742e77198219 P cfae6937c171468db2b68d87b1149dea [LEADER]: Connected to new peer: Peer: permanent_uuid: "3fd44bea82cc4e89bfc9ada83a1fd302" member_type: VOTER last_known_addr { host: "127.11.102.1" port: 37171 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 06:09:43.666071 27551 consensus_queue.cc:1035] T 11aae1fd98d748b89baa742e77198219 P cfae6937c171468db2b68d87b1149dea [LEADER]: Connected to new peer: Peer: permanent_uuid: "5600989c2f634c4cb630839afa98c1a3" member_type: VOTER last_known_addr { host: "127.11.102.3" port: 43283 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 06:09:43.673303 27556 mvcc.cc:205] Tried to move back new op lower bound from 7298808969884569600 to 7298808969752846336. Current Snapshot: MvccSnapshot[applied={T|T < 7298808969884569600}]
I20260620 06:09:43.674714 27554 mvcc.cc:205] Tried to move back new op lower bound from 7298808969884569600 to 7298808969752846336. Current Snapshot: MvccSnapshot[applied={T|T < 7298808969884569600}]
I20260620 06:09:43.682559 27062 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33372:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 06:09:43.682754 27062 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33372:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 06:09:43.685189 27062 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 08cf14c4a270431082d354a7a84ab2fd: Sending DeleteTablet for 3 replicas of tablet 11aae1fd98d748b89baa742e77198219
I20260620 06:09:43.685942 27457 tablet_service.cc:1514] Processing DeleteTablet for tablet 11aae1fd98d748b89baa742e77198219 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 06:09:43 UTC) from {username='kudu', principal='kudu/127.11.102.62@KRBTEST.COM'} at 127.0.0.1:55586
I20260620 06:09:43.686017 11672 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9HMp43/build/debug/bin/kudu with pid 27126
I20260620 06:09:43.686175 27323 tablet_service.cc:1514] Processing DeleteTablet for tablet 11aae1fd98d748b89baa742e77198219 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 06:09:43 UTC) from {username='kudu', principal='kudu/127.11.102.62@KRBTEST.COM'} at 127.0.0.1:41630
I20260620 06:09:43.687089 27563 tablet_replica.cc:330] T 11aae1fd98d748b89baa742e77198219 P 5600989c2f634c4cb630839afa98c1a3: stopping tablet replica
I20260620 06:09:43.687407 27563 raft_consensus.cc:2229] T 11aae1fd98d748b89baa742e77198219 P 5600989c2f634c4cb630839afa98c1a3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 06:09:43.687757 27563 raft_consensus.cc:2258] T 11aae1fd98d748b89baa742e77198219 P 5600989c2f634c4cb630839afa98c1a3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 06:09:43.688006 27564 tablet_replica.cc:330] T 11aae1fd98d748b89baa742e77198219 P cfae6937c171468db2b68d87b1149dea: stopping tablet replica
I20260620 06:09:43.688311 27564 raft_consensus.cc:2229] T 11aae1fd98d748b89baa742e77198219 P cfae6937c171468db2b68d87b1149dea [term 1 LEADER]: Raft consensus shutting down.
I20260620 06:09:43.688730 27564 raft_consensus.cc:2258] T 11aae1fd98d748b89baa742e77198219 P cfae6937c171468db2b68d87b1149dea [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 06:09:43.690379 27563 ts_tablet_manager.cc:1905] T 11aae1fd98d748b89baa742e77198219 P 5600989c2f634c4cb630839afa98c1a3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 06:09:43.690382 27564 ts_tablet_manager.cc:1905] T 11aae1fd98d748b89baa742e77198219 P cfae6937c171468db2b68d87b1149dea: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 06:09:43.693329 27563 ts_tablet_manager.cc:1918] T 11aae1fd98d748b89baa742e77198219 P 5600989c2f634c4cb630839afa98c1a3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 06:09:43.693491 27563 log.cc:1196] T 11aae1fd98d748b89baa742e77198219 P 5600989c2f634c4cb630839afa98c1a3: Deleting WAL directory at /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935603120653-11672-0/minicluster-data/ts-2/wal/wals/11aae1fd98d748b89baa742e77198219
W20260620 06:09:43.693711 27050 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.11.102.1:37171 (error 108)
I20260620 06:09:43.693894 27563 ts_tablet_manager.cc:1939] T 11aae1fd98d748b89baa742e77198219 P 5600989c2f634c4cb630839afa98c1a3: Deleting consensus metadata
I20260620 06:09:43.693981 11672 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9HMp43/build/debug/bin/kudu with pid 27260
I20260620 06:09:43.696456 27538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:09:43.696174 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.102.1:37171 (local address 127.0.0.1:44960)
0620 06:09:43.696302 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0620 06:09:43.696375 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.102.1:37171: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260620 06:09:43.696642 27050 catalog_manager.cc:4584] TS 3fd44bea82cc4e89bfc9ada83a1fd302 (127.11.102.1:37171): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 11aae1fd98d748b89baa742e77198219: Network error: Client connection negotiation failed: client connection to 127.11.102.1:37171: connect: Connection refused (error 111)
I20260620 06:09:43.696687 27049 catalog_manager.cc:4841] TS 5600989c2f634c4cb630839afa98c1a3 (127.11.102.3:43283): tablet 11aae1fd98d748b89baa742e77198219 (table test-table [id=e26f28e3aa7045758855c1c72065084c]) successfully deleted
I20260620 06:09:43.699625 11672 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9HMp43/build/debug/bin/kudu with pid 27394
I20260620 06:09:43.700655 27538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:09:43.700338 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.102.2:40107 (local address 127.0.0.1:41648)
0620 06:09:43.700484 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0620 06:09:43.700550 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.102.2:40107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260620 06:09:43.705233 11672 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9HMp43/build/debug/bin/kudu with pid 27032
I20260620 06:10:03.711585 11672 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9HMp43/build/debug/bin/kudu
/tmp/dist-test-task9HMp43/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935603120653-11672-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935603120653-11672-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935603120653-11672-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935603120653-11672-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.102.62:41831
--webserver_interface=127.11.102.62
--webserver_port=41901
--keytab_file=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935603120653-11672-0/krb5kdc/kudu_127.11.102.62.keytab
--principal=kudu/127.11.102.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.102.20:38439
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.102.62:41831
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935603120653-11672-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935603120653-11672-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935603120653-11672-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 06:10:03.820714 27566 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:10:03.820981 27566 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:10:03.821022 27566 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:10:03.824903 27566 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260620 06:10:03.824950 27566 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260620 06:10:03.824966 27566 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260620 06:10:03.824981 27566 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 06:10:03.824999 27566 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:10:03.825017 27566 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:10:03.825032 27566 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 06:10:03.825047 27566 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 06:10:03.830406 27566 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.102.20:38439
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935603120653-11672-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935603120653-11672-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.11.102.62:41831
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935603120653-11672-0/krb5kdc/kudu_127.11.102.62.keytab
--principal=kudu/127.11.102.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.102.62:41831
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935603120653-11672-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.102.62
--webserver_port=41901
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27566
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935603120653-11672-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 05:28:48 UTC on ccycloud.quasar-fixhms.root.comops.site
build id 4457020
I20260620 06:10:03.831430 27566 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:10:03.832422 27566 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 06:10:03.847556 27573 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:10:03.847556 27575 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:10:03.847836 27572 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:10:03.847597 27566 server_base.cc:1034] running on GCE node
I20260620 06:10:03.848459 27566 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:10:03.849624 27566 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:10:03.850787 27566 hybrid_clock.cc:648] HybridClock initialized: now 1781935803850763 us; error 36 us; skew 500 ppm
Jun 20 06:10:03 dist-test-slave-9hm5 krb5kdc[27019](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935803, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.102.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 06:10:03.853569 27566 init.cc:377] Logged in from keytab as kudu/127.11.102.62@KRBTEST.COM (short username kudu)
I20260620 06:10:03.854539 27566 webserver.cc:460] Webserver started at http://127.11.102.62:41901/ using document root <none> and password file <none>
I20260620 06:10:03.855187 27566 fs_manager.cc:359] Metadata directory not provided
I20260620 06:10:03.855285 27566 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:10:03.859326 27566 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260620 06:10:03.861529 27582 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:10:03.862691 27566 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260620 06:10:03.862834 27566 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935603120653-11672-0/minicluster-data/master-0/data,/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935603120653-11672-0/minicluster-data/master-0/wal
uuid: "08cf14c4a270431082d354a7a84ab2fd"
format_stamp: "Formatted at 2026-06-20 06:09:42 on dist-test-slave-9hm5"
server_key: "30abf25c0b3089745fb551a08427a3b0"
server_key_iv: "5d2f4a95c2affcce55c4ce2dcb4b6639"
server_key_version: "encryptionkey@0"
I20260620 06:10:03.863461 27566 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935603120653-11672-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935603120653-11672-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935603120653-11672-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:10:03.878680 27566 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:10:03.881574 27566 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:10:03.881783 27566 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:10:03.888923 27566 rpc_server.cc:307] RPC server started. Bound to: 127.11.102.62:41831
I20260620 06:10:03.889209 27634 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.102.62:41831 every 8 connection(s)
I20260620 06:10:03.890024 27566 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935603120653-11672-0/minicluster-data/master-0/data/info.pb
I20260620 06:10:03.894338 27635 sys_catalog.cc:263] Verifying existing consensus state
I20260620 06:10:03.896026 27635 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 08cf14c4a270431082d354a7a84ab2fd: Bootstrap starting.
I20260620 06:10:03.898350 11672 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9HMp43/build/debug/bin/kudu as pid 27566
I20260620 06:10:03.898800 11672 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9HMp43/build/debug/bin/kudu
/tmp/dist-test-task9HMp43/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935603120653-11672-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935603120653-11672-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935603120653-11672-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935603120653-11672-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.102.1:37171
--local_ip_for_outbound_sockets=127.11.102.1
--tserver_master_addrs=127.11.102.62:41831
--webserver_port=34149
--webserver_interface=127.11.102.1
--keytab_file=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935603120653-11672-0/krb5kdc/kudu_127.11.102.1.keytab
--principal=kudu/127.11.102.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.102.20:38439
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935603120653-11672-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935603120653-11672-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935603120653-11672-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 06:10:03.918685 27635 log.cc:824] T 00000000000000000000000000000000 P 08cf14c4a270431082d354a7a84ab2fd: Log is configured to *not* fsync() on all Append() calls
I20260620 06:10:03.929843 27635 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 08cf14c4a270431082d354a7a84ab2fd: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 06:10:03.930269 27635 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 08cf14c4a270431082d354a7a84ab2fd: Bootstrap complete.
I20260620 06:10:03.935743 27635 raft_consensus.cc:348] T 00000000000000000000000000000000 P 08cf14c4a270431082d354a7a84ab2fd [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "08cf14c4a270431082d354a7a84ab2fd" member_type: VOTER last_known_addr { host: "127.11.102.62" port: 41831 } }
I20260620 06:10:03.936244 27635 raft_consensus.cc:729] T 00000000000000000000000000000000 P 08cf14c4a270431082d354a7a84ab2fd [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 08cf14c4a270431082d354a7a84ab2fd, State: Initialized, Role: FOLLOWER
I20260620 06:10:03.936779 27635 consensus_queue.cc:260] T 00000000000000000000000000000000 P 08cf14c4a270431082d354a7a84ab2fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "08cf14c4a270431082d354a7a84ab2fd" member_type: VOTER last_known_addr { host: "127.11.102.62" port: 41831 } }
I20260620 06:10:03.936919 27635 raft_consensus.cc:388] T 00000000000000000000000000000000 P 08cf14c4a270431082d354a7a84ab2fd [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 06:10:03.936986 27635 raft_consensus.cc:482] T 00000000000000000000000000000000 P 08cf14c4a270431082d354a7a84ab2fd [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 06:10:03.937090 27635 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 08cf14c4a270431082d354a7a84ab2fd [term 1 FOLLOWER]: Advancing to term 2
I20260620 06:10:03.939455 27635 raft_consensus.cc:504] T 00000000000000000000000000000000 P 08cf14c4a270431082d354a7a84ab2fd [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "08cf14c4a270431082d354a7a84ab2fd" member_type: VOTER last_known_addr { host: "127.11.102.62" port: 41831 } }
I20260620 06:10:03.939839 27635 leader_election.cc:302] T 00000000000000000000000000000000 P 08cf14c4a270431082d354a7a84ab2fd [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 08cf14c4a270431082d354a7a84ab2fd; no voters:
I20260620 06:10:03.940186 27635 leader_election.cc:288] T 00000000000000000000000000000000 P 08cf14c4a270431082d354a7a84ab2fd [CANDIDATE]: Term 2 election: Requested vote from peers
I20260620 06:10:03.940526 27639 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 08cf14c4a270431082d354a7a84ab2fd [term 2 FOLLOWER]: Leader election won for term 2
I20260620 06:10:03.940871 27639 raft_consensus.cc:686] T 00000000000000000000000000000000 P 08cf14c4a270431082d354a7a84ab2fd [term 2 LEADER]: Becoming Leader. State: Replica: 08cf14c4a270431082d354a7a84ab2fd, State: Running, Role: LEADER
I20260620 06:10:03.941224 27639 consensus_queue.cc:237] T 00000000000000000000000000000000 P 08cf14c4a270431082d354a7a84ab2fd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "08cf14c4a270431082d354a7a84ab2fd" member_type: VOTER last_known_addr { host: "127.11.102.62" port: 41831 } }
I20260620 06:10:03.941514 27635 sys_catalog.cc:564] T 00000000000000000000000000000000 P 08cf14c4a270431082d354a7a84ab2fd [sys.catalog]: configured and running, proceeding with master startup.
I20260620 06:10:03.942757 27641 sys_catalog.cc:455] T 00000000000000000000000000000000 P 08cf14c4a270431082d354a7a84ab2fd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 08cf14c4a270431082d354a7a84ab2fd. Latest consensus state: current_term: 2 leader_uuid: "08cf14c4a270431082d354a7a84ab2fd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "08cf14c4a270431082d354a7a84ab2fd" member_type: VOTER last_known_addr { host: "127.11.102.62" port: 41831 } } }
I20260620 06:10:03.942782 27640 sys_catalog.cc:455] T 00000000000000000000000000000000 P 08cf14c4a270431082d354a7a84ab2fd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "08cf14c4a270431082d354a7a84ab2fd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "08cf14c4a270431082d354a7a84ab2fd" member_type: VOTER last_known_addr { host: "127.11.102.62" port: 41831 } } }
I20260620 06:10:03.942901 27640 sys_catalog.cc:458] T 00000000000000000000000000000000 P 08cf14c4a270431082d354a7a84ab2fd [sys.catalog]: This master's current role is: LEADER
I20260620 06:10:03.942901 27641 sys_catalog.cc:458] T 00000000000000000000000000000000 P 08cf14c4a270431082d354a7a84ab2fd [sys.catalog]: This master's current role is: LEADER
I20260620 06:10:03.943327 27649 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 06:10:03.947398 27649 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 06:10:03.947988 27649 catalog_manager.cc:1259] Loaded cluster ID: c5bf0008db3641f5aa12f375b47b3a94
I20260620 06:10:03.948064 27649 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 06:10:03.949802 27649 catalog_manager.cc:1504] Loading token signing keys...
I20260620 06:10:03.950404 27649 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 08cf14c4a270431082d354a7a84ab2fd: Loaded TSK: 0
I20260620 06:10:03.962178 27649 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 08cf14c4a270431082d354a7a84ab2fd: Generated new TSK 1
I20260620 06:10:03.962458 27649 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260620 06:10:04.031860 27637 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:10:04.032094 27637 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:10:04.032135 27637 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:10:04.036204 27637 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:10:04.036269 27637 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:10:04.036329 27637 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.102.1
I20260620 06:10:04.041836 27637 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.102.20:38439
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935603120653-11672-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935603120653-11672-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935603120653-11672-0/krb5kdc/kudu_127.11.102.1.keytab
--principal=kudu/127.11.102.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.102.1:37171
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935603120653-11672-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.11.102.1
--webserver_port=34149
--tserver_master_addrs=127.11.102.62:41831
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27637
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.102.1
--log_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935603120653-11672-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 05:28:48 UTC on ccycloud.quasar-fixhms.root.comops.site
build id 4457020
I20260620 06:10:04.042979 27637 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:10:04.044082 27637 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 06:10:04.053773 27662 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:10:04.053791 27663 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:10:04.053841 27665 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:10:04.054313 27637 server_base.cc:1034] running on GCE node
I20260620 06:10:04.054723 27637 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:10:04.055473 27637 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:10:04.056681 27637 hybrid_clock.cc:648] HybridClock initialized: now 1781935804056652 us; error 48 us; skew 500 ppm
Jun 20 06:10:04 dist-test-slave-9hm5 krb5kdc[27019](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935804, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.102.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 06:10:04.059482 27637 init.cc:377] Logged in from keytab as kudu/127.11.102.1@KRBTEST.COM (short username kudu)
I20260620 06:10:04.060416 27637 webserver.cc:460] Webserver started at http://127.11.102.1:34149/ using document root <none> and password file <none>
I20260620 06:10:04.060990 27637 fs_manager.cc:359] Metadata directory not provided
I20260620 06:10:04.061064 27637 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:10:04.064908 27637 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.001s sys 0.002s
I20260620 06:10:04.067466 27672 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:10:04.068785 27637 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 06:10:04.068933 27637 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935603120653-11672-0/minicluster-data/ts-0/data,/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935603120653-11672-0/minicluster-data/ts-0/wal
uuid: "3fd44bea82cc4e89bfc9ada83a1fd302"
format_stamp: "Formatted at 2026-06-20 06:09:42 on dist-test-slave-9hm5"
server_key: "87f03c04be86da2aa07d685cba61f524"
server_key_iv: "0050b961ddf0b580701336a7fd81ae80"
server_key_version: "encryptionkey@0"
I20260620 06:10:04.069473 27637 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935603120653-11672-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935603120653-11672-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935603120653-11672-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:10:04.085515 27637 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:10:04.088954 27637 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:10:04.089161 27637 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:10:04.089763 27637 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 06:10:04.090886 27679 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260620 06:10:04.094985 27637 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260620 06:10:04.095049 27637 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.004s user 0.001s sys 0.000s
I20260620 06:10:04.095098 27637 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260620 06:10:04.096922 27637 ts_tablet_manager.cc:610] Registered 1 tablets
I20260620 06:10:04.096982 27637 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.001s sys 0.000s
I20260620 06:10:04.097239 27679 tablet_bootstrap.cc:492] T 11aae1fd98d748b89baa742e77198219 P 3fd44bea82cc4e89bfc9ada83a1fd302: Bootstrap starting.
I20260620 06:10:04.108062 27637 rpc_server.cc:307] RPC server started. Bound to: 127.11.102.1:37171
I20260620 06:10:04.108089 27786 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.102.1:37171 every 8 connection(s)
I20260620 06:10:04.109326 27637 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935603120653-11672-0/minicluster-data/ts-0/data/info.pb
I20260620 06:10:04.116619 11672 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9HMp43/build/debug/bin/kudu as pid 27637
I20260620 06:10:04.117143 11672 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9HMp43/build/debug/bin/kudu
/tmp/dist-test-task9HMp43/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935603120653-11672-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935603120653-11672-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935603120653-11672-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935603120653-11672-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.102.2:40107
--local_ip_for_outbound_sockets=127.11.102.2
--tserver_master_addrs=127.11.102.62:41831
--webserver_port=42091
--webserver_interface=127.11.102.2
--keytab_file=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935603120653-11672-0/krb5kdc/kudu_127.11.102.2.keytab
--principal=kudu/127.11.102.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.102.20:38439
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935603120653-11672-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935603120653-11672-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935603120653-11672-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 06:10:04.117801 27679 log.cc:824] T 11aae1fd98d748b89baa742e77198219 P 3fd44bea82cc4e89bfc9ada83a1fd302: Log is configured to *not* fsync() on all Append() calls
Jun 20 06:10:04 dist-test-slave-9hm5 krb5kdc[27019](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935804, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.102.1@KRBTEST.COM for kudu/127.11.102.62@KRBTEST.COM
I20260620 06:10:04.125881 27679 tablet_bootstrap.cc:492] T 11aae1fd98d748b89baa742e77198219 P 3fd44bea82cc4e89bfc9ada83a1fd302: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 06:10:04.126330 27679 tablet_bootstrap.cc:492] T 11aae1fd98d748b89baa742e77198219 P 3fd44bea82cc4e89bfc9ada83a1fd302: Bootstrap complete.
I20260620 06:10:04.127681 27679 ts_tablet_manager.cc:1397] T 11aae1fd98d748b89baa742e77198219 P 3fd44bea82cc4e89bfc9ada83a1fd302: Time spent bootstrapping tablet: real 0.031s user 0.022s sys 0.008s
I20260620 06:10:04.128257 27789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:10:04.111508 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.102.1:46175 (local address 127.11.102.62:41831)
0620 06:10:04.111913 (+ 405us) server_negotiation.cc:206] Beginning negotiation
0620 06:10:04.111934 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0620 06:10:04.112266 (+ 332us) server_negotiation.cc:407] Connection header received
0620 06:10:04.113387 (+ 1121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:10:04.113398 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:10:04.113691 (+ 293us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:10:04.114013 (+ 322us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:10:04.115127 (+ 1114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:10:04.116172 (+ 1045us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:10:04.117972 (+ 1800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:10:04.118765 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:10:04.118953 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:10:04.122035 (+ 3082us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:10:04.122042 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:10:04.122056 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:10:04.122089 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:10:04.124334 (+ 2245us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:10:04.125025 (+ 691us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:10:04.125031 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:10:04.125036 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:10:04.125188 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:10:04.125631 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:10:04.125634 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:10:04.125636 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:10:04.126135 (+ 499us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:10:04.126279 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0620 06:10:04.126982 (+ 703us) server_negotiation.cc:299] Negotiation successful
0620 06:10:04.127293 (+ 311us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":260,"thread_start_us":172,"threads_started":1}
I20260620 06:10:04.128368 27790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:10:04.111530 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.102.62:41831 (local address 127.11.102.1:46175)
0620 06:10:04.112101 (+ 571us) negotiation.cc:107] Waiting for socket to connect
0620 06:10:04.112144 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0620 06:10:04.112958 (+ 814us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:10:04.114073 (+ 1115us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:10:04.114081 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:10:04.114495 (+ 414us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:10:04.114941 (+ 446us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:10:04.114953 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:10:04.116338 (+ 1385us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:10:04.116343 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:10:04.117794 (+ 1451us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:10:04.117809 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:10:04.118917 (+ 1108us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:10:04.118921 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:10:04.119149 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:10:04.119811 (+ 662us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:10:04.119830 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:10:04.121808 (+ 1978us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:10:04.124470 (+ 2662us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:10:04.124480 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:10:04.124496 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:10:04.124889 (+ 393us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:10:04.125304 (+ 415us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:10:04.125310 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:10:04.125314 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:10:04.125512 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:10:04.126271 (+ 759us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:10:04.126281 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:10:04.126617 (+ 336us) client_negotiation.cc:769] Sending connection context
0620 06:10:04.126845 (+ 228us) client_negotiation.cc:240] Negotiation successful
0620 06:10:04.127238 (+ 393us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":364,"thread_start_us":170,"threads_started":1}
I20260620 06:10:04.130435 27679 raft_consensus.cc:348] T 11aae1fd98d748b89baa742e77198219 P 3fd44bea82cc4e89bfc9ada83a1fd302 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5600989c2f634c4cb630839afa98c1a3" member_type: VOTER last_known_addr { host: "127.11.102.3" port: 43283 } } peers { permanent_uuid: "3fd44bea82cc4e89bfc9ada83a1fd302" member_type: VOTER last_known_addr { host: "127.11.102.1" port: 37171 } } peers { permanent_uuid: "cfae6937c171468db2b68d87b1149dea" member_type: VOTER last_known_addr { host: "127.11.102.2" port: 40107 } }
I20260620 06:10:04.130981 27679 raft_consensus.cc:729] T 11aae1fd98d748b89baa742e77198219 P 3fd44bea82cc4e89bfc9ada83a1fd302 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3fd44bea82cc4e89bfc9ada83a1fd302, State: Initialized, Role: FOLLOWER
I20260620 06:10:04.131073 27787 heartbeater.cc:347] Connected to a master server at 127.11.102.62:41831
I20260620 06:10:04.131301 27787 heartbeater.cc:464] Registering TS with master...
I20260620 06:10:04.131522 27679 consensus_queue.cc:260] T 11aae1fd98d748b89baa742e77198219 P 3fd44bea82cc4e89bfc9ada83a1fd302 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5600989c2f634c4cb630839afa98c1a3" member_type: VOTER last_known_addr { host: "127.11.102.3" port: 43283 } } peers { permanent_uuid: "3fd44bea82cc4e89bfc9ada83a1fd302" member_type: VOTER last_known_addr { host: "127.11.102.1" port: 37171 } } peers { permanent_uuid: "cfae6937c171468db2b68d87b1149dea" member_type: VOTER last_known_addr { host: "127.11.102.2" port: 40107 } }
I20260620 06:10:04.131860 27787 heartbeater.cc:511] Master 127.11.102.62:41831 requested a full tablet report, sending...
I20260620 06:10:04.132529 27679 ts_tablet_manager.cc:1428] T 11aae1fd98d748b89baa742e77198219 P 3fd44bea82cc4e89bfc9ada83a1fd302: Time spent starting tablet: real 0.005s user 0.001s sys 0.003s
I20260620 06:10:04.134369 27599 ts_manager.cc:194] Registered new tserver with Master: 3fd44bea82cc4e89bfc9ada83a1fd302 (127.11.102.1:37171)
I20260620 06:10:04.137696 27599 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.102.1@KRBTEST.COM'} at 127.11.102.1:46175
I20260620 06:10:04.140403 27787 heartbeater.cc:503] Master 127.11.102.62:41831 was elected leader, sending a full tablet report...
Jun 20 06:10:04 dist-test-slave-9hm5 krb5kdc[27019](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935803, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.102.62@KRBTEST.COM for kudu/127.11.102.1@KRBTEST.COM
I20260620 06:10:04.147558 27794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:10:04.137112 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.102.1:37171 (local address 127.0.0.1:53424)
0620 06:10:04.137845 (+ 733us) negotiation.cc:107] Waiting for socket to connect
0620 06:10:04.137888 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0620 06:10:04.138022 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:10:04.138573 (+ 551us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:10:04.138579 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:10:04.138946 (+ 367us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:10:04.139047 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:10:04.139054 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:10:04.140153 (+ 1099us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:10:04.140158 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:10:04.140974 (+ 816us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:10:04.140985 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:10:04.141647 (+ 662us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:10:04.141651 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:10:04.141758 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:10:04.142386 (+ 628us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:10:04.142402 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:10:04.143984 (+ 1582us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:10:04.145856 (+ 1872us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:10:04.145866 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:10:04.145871 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:10:04.146095 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:10:04.146388 (+ 293us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:10:04.146392 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:10:04.146394 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:10:04.146481 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:10:04.146933 (+ 452us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:10:04.146939 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:10:04.147057 (+ 118us) client_negotiation.cc:769] Sending connection context
0620 06:10:04.147193 (+ 136us) client_negotiation.cc:240] Negotiation successful
0620 06:10:04.147384 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":568,"thread_start_us":117,"threads_started":1}
I20260620 06:10:04.147578 27795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:10:04.137204 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53424 (local address 127.11.102.1:37171)
0620 06:10:04.137579 (+ 375us) server_negotiation.cc:206] Beginning negotiation
0620 06:10:04.137591 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 06:10:04.137988 (+ 397us) server_negotiation.cc:407] Connection header received
0620 06:10:04.138170 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:10:04.138177 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:10:04.138381 (+ 204us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:10:04.138509 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:10:04.139171 (+ 662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:10:04.140047 (+ 876us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:10:04.141100 (+ 1053us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:10:04.141525 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:10:04.141635 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:10:04.144123 (+ 2488us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:10:04.144132 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:10:04.144144 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:10:04.144168 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:10:04.145727 (+ 1559us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:10:04.146208 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:10:04.146214 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:10:04.146216 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:10:04.146294 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:10:04.146569 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:10:04.146574 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:10:04.146576 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:10:04.146801 (+ 225us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:10:04.146946 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0620 06:10:04.147282 (+ 336us) server_negotiation.cc:299] Negotiation successful
0620 06:10:04.147436 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":269,"thread_start_us":100,"threads_started":1}
I20260620 06:10:04.149174 27721 tablet_service.cc:1514] Processing DeleteTablet for tablet 11aae1fd98d748b89baa742e77198219 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 06:09:43 UTC) from {username='kudu', principal='kudu/127.11.102.62@KRBTEST.COM'} at 127.0.0.1:53424
I20260620 06:10:04.150151 27796 tablet_replica.cc:330] stopping tablet replica
I20260620 06:10:04.150442 27796 raft_consensus.cc:2229] T 11aae1fd98d748b89baa742e77198219 P 3fd44bea82cc4e89bfc9ada83a1fd302 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 06:10:04.150686 27796 raft_consensus.cc:2258] T 11aae1fd98d748b89baa742e77198219 P 3fd44bea82cc4e89bfc9ada83a1fd302 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 06:10:04.151674 27796 ts_tablet_manager.cc:1905] T 11aae1fd98d748b89baa742e77198219 P 3fd44bea82cc4e89bfc9ada83a1fd302: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 06:10:04.155242 27796 ts_tablet_manager.cc:1918] T 11aae1fd98d748b89baa742e77198219 P 3fd44bea82cc4e89bfc9ada83a1fd302: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 06:10:04.155359 27796 log.cc:1196] T 11aae1fd98d748b89baa742e77198219 P 3fd44bea82cc4e89bfc9ada83a1fd302: Deleting WAL directory at /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935603120653-11672-0/minicluster-data/ts-0/wal/wals/11aae1fd98d748b89baa742e77198219
I20260620 06:10:04.155745 27796 ts_tablet_manager.cc:1939] T 11aae1fd98d748b89baa742e77198219 P 3fd44bea82cc4e89bfc9ada83a1fd302: Deleting consensus metadata
I20260620 06:10:04.157480 27585 catalog_manager.cc:4841] TS 3fd44bea82cc4e89bfc9ada83a1fd302 (127.11.102.1:37171): tablet 11aae1fd98d748b89baa742e77198219 (table test-table [id=e26f28e3aa7045758855c1c72065084c]) successfully deleted
W20260620 06:10:04.235657 27792 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:10:04.235937 27792 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:10:04.235982 27792 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:10:04.239995 27792 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:10:04.240065 27792 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:10:04.240124 27792 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.102.2
I20260620 06:10:04.245677 27792 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.102.20:38439
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935603120653-11672-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935603120653-11672-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935603120653-11672-0/krb5kdc/kudu_127.11.102.2.keytab
--principal=kudu/127.11.102.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.102.2:40107
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935603120653-11672-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.11.102.2
--webserver_port=42091
--tserver_master_addrs=127.11.102.62:41831
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27792
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.102.2
--log_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935603120653-11672-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 05:28:48 UTC on ccycloud.quasar-fixhms.root.comops.site
build id 4457020
I20260620 06:10:04.246685 27792 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:10:04.247742 27792 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 06:10:04.257457 27801 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:10:04.257442 27804 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:10:04.257688 27792 server_base.cc:1034] running on GCE node
W20260620 06:10:04.257849 27802 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:10:04.258289 27792 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:10:04.258972 27792 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:10:04.260172 27792 hybrid_clock.cc:648] HybridClock initialized: now 1781935804260136 us; error 49 us; skew 500 ppm
Jun 20 06:10:04 dist-test-slave-9hm5 krb5kdc[27019](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935804, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.102.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 06:10:04.263149 27792 init.cc:377] Logged in from keytab as kudu/127.11.102.2@KRBTEST.COM (short username kudu)
I20260620 06:10:04.264103 27792 webserver.cc:460] Webserver started at http://127.11.102.2:42091/ using document root <none> and password file <none>
I20260620 06:10:04.264631 27792 fs_manager.cc:359] Metadata directory not provided
I20260620 06:10:04.264696 27792 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:10:04.268662 27792 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 06:10:04.272452 27811 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:10:04.273792 27792 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 06:10:04.273903 27792 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935603120653-11672-0/minicluster-data/ts-1/data,/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935603120653-11672-0/minicluster-data/ts-1/wal
uuid: "cfae6937c171468db2b68d87b1149dea"
format_stamp: "Formatted at 2026-06-20 06:09:43 on dist-test-slave-9hm5"
server_key: "4e9bccdc31a644917c436f65b57a1180"
server_key_iv: "bf1fa8eb3d43a242ea056778541df575"
server_key_version: "encryptionkey@0"
I20260620 06:10:04.274462 27792 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935603120653-11672-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935603120653-11672-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935603120653-11672-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:10:04.302640 27792 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:10:04.306324 27792 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:10:04.306540 27792 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:10:04.307344 27792 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 06:10:04.308460 27818 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
W20260620 06:10:04.312175 27818 ts_tablet_manager.cc:1852] T 11aae1fd98d748b89baa742e77198219 P cfae6937c171468db2b68d87b1149dea: Tablet Manager startup: Rolling forward tablet deletion of type TABLET_DATA_DELETED
I20260620 06:10:04.312547 27818 ts_tablet_manager.cc:1905] T 11aae1fd98d748b89baa742e77198219 P cfae6937c171468db2b68d87b1149dea: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 06:10:04.316291 27818 ts_tablet_manager.cc:1918] T 11aae1fd98d748b89baa742e77198219 P cfae6937c171468db2b68d87b1149dea: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 06:10:04.316385 27818 log.cc:1196] T 11aae1fd98d748b89baa742e77198219 P cfae6937c171468db2b68d87b1149dea: Deleting WAL directory at /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935603120653-11672-0/minicluster-data/ts-1/wal/wals/11aae1fd98d748b89baa742e77198219
I20260620 06:10:04.316726 27818 ts_tablet_manager.cc:1939] T 11aae1fd98d748b89baa742e77198219 P cfae6937c171468db2b68d87b1149dea: Deleting consensus metadata
I20260620 06:10:04.317040 27792 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 0 live tablets)
I20260620 06:10:04.317116 27792 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.009s user 0.000s sys 0.000s
I20260620 06:10:04.317217 27792 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 06:10:04.317261 27792 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 06:10:04.326409 27792 rpc_server.cc:307] RPC server started. Bound to: 127.11.102.2:40107
I20260620 06:10:04.326442 27925 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.102.2:40107 every 8 connection(s)
I20260620 06:10:04.327289 27792 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935603120653-11672-0/minicluster-data/ts-1/data/info.pb
I20260620 06:10:04.335672 11672 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9HMp43/build/debug/bin/kudu as pid 27792
I20260620 06:10:04.336203 11672 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9HMp43/build/debug/bin/kudu
/tmp/dist-test-task9HMp43/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935603120653-11672-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935603120653-11672-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935603120653-11672-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935603120653-11672-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.102.3:43283
--local_ip_for_outbound_sockets=127.11.102.3
--tserver_master_addrs=127.11.102.62:41831
--webserver_port=42325
--webserver_interface=127.11.102.3
--keytab_file=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935603120653-11672-0/krb5kdc/kudu_127.11.102.3.keytab
--principal=kudu/127.11.102.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.102.20:38439
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935603120653-11672-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935603120653-11672-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935603120653-11672-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 06:10:04 dist-test-slave-9hm5 krb5kdc[27019](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935804, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.102.2@KRBTEST.COM for kudu/127.11.102.62@KRBTEST.COM
I20260620 06:10:04.341305 27789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:10:04.329066 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.102.2:58299 (local address 127.11.102.62:41831)
0620 06:10:04.329245 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0620 06:10:04.329249 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 06:10:04.330101 (+ 852us) server_negotiation.cc:407] Connection header received
0620 06:10:04.331023 (+ 922us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:10:04.331027 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:10:04.331080 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:10:04.331189 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:10:04.332374 (+ 1185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:10:04.332913 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:10:04.334023 (+ 1110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:10:04.334257 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:10:04.334368 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:10:04.337148 (+ 2780us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:10:04.337152 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:10:04.337155 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:10:04.337185 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:10:04.338623 (+ 1438us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:10:04.339198 (+ 575us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:10:04.339205 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:10:04.339210 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:10:04.339340 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:10:04.339928 (+ 588us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:10:04.339935 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:10:04.339939 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:10:04.340203 (+ 264us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:10:04.340318 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0620 06:10:04.341073 (+ 755us) server_negotiation.cc:299] Negotiation successful
0620 06:10:04.341184 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260620 06:10:04.342533 27928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:10:04.329475 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.102.62:41831 (local address 127.11.102.2:58299)
0620 06:10:04.329932 (+ 457us) negotiation.cc:107] Waiting for socket to connect
0620 06:10:04.329977 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0620 06:10:04.330820 (+ 843us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:10:04.331367 (+ 547us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:10:04.331374 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:10:04.331776 (+ 402us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:10:04.332232 (+ 456us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:10:04.332245 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:10:04.333042 (+ 797us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:10:04.333050 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:10:04.333899 (+ 849us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:10:04.333908 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:10:04.334413 (+ 505us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:10:04.334417 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:10:04.334577 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:10:04.335201 (+ 624us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:10:04.335217 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:10:04.336985 (+ 1768us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:10:04.338759 (+ 1774us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:10:04.338764 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:10:04.338778 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:10:04.339028 (+ 250us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:10:04.339449 (+ 421us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:10:04.339452 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:10:04.339453 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:10:04.339578 (+ 125us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:10:04.340394 (+ 816us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:10:04.340402 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:10:04.340737 (+ 335us) client_negotiation.cc:769] Sending connection context
0620 06:10:04.340966 (+ 229us) client_negotiation.cc:240] Negotiation successful
0620 06:10:04.341349 (+ 383us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":255,"thread_start_us":109,"threads_started":1}
I20260620 06:10:04.343972 27926 heartbeater.cc:347] Connected to a master server at 127.11.102.62:41831
I20260620 06:10:04.344252 27926 heartbeater.cc:464] Registering TS with master...
I20260620 06:10:04.344724 27926 heartbeater.cc:511] Master 127.11.102.62:41831 requested a full tablet report, sending...
I20260620 06:10:04.345749 27599 ts_manager.cc:194] Registered new tserver with Master: cfae6937c171468db2b68d87b1149dea (127.11.102.2:40107)
I20260620 06:10:04.346309 27599 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.102.2@KRBTEST.COM'} at 127.11.102.2:58299
W20260620 06:10:04.449532 27929 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:10:04.449832 27929 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:10:04.449942 27929 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:10:04.454044 27929 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:10:04.454130 27929 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:10:04.454218 27929 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.102.3
I20260620 06:10:04.459800 27929 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.102.20:38439
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935603120653-11672-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935603120653-11672-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935603120653-11672-0/krb5kdc/kudu_127.11.102.3.keytab
--principal=kudu/127.11.102.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.102.3:43283
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935603120653-11672-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.11.102.3
--webserver_port=42325
--tserver_master_addrs=127.11.102.62:41831
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27929
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.102.3
--log_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935603120653-11672-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 05:28:48 UTC on ccycloud.quasar-fixhms.root.comops.site
build id 4457020
I20260620 06:10:04.460873 27929 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:10:04.461958 27929 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 06:10:04.471266 27935 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:10:04.471432 27934 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:10:04.471273 27937 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:10:04.471412 27929 server_base.cc:1034] running on GCE node
I20260620 06:10:04.471947 27929 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:10:04.472491 27929 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:10:04.473697 27929 hybrid_clock.cc:648] HybridClock initialized: now 1781935804473681 us; error 24 us; skew 500 ppm
Jun 20 06:10:04 dist-test-slave-9hm5 krb5kdc[27019](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935804, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.102.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 06:10:04.476471 27929 init.cc:377] Logged in from keytab as kudu/127.11.102.3@KRBTEST.COM (short username kudu)
I20260620 06:10:04.477281 27929 webserver.cc:460] Webserver started at http://127.11.102.3:42325/ using document root <none> and password file <none>
I20260620 06:10:04.477842 27929 fs_manager.cc:359] Metadata directory not provided
I20260620 06:10:04.477934 27929 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:10:04.481700 27929 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.001s
I20260620 06:10:04.483704 27944 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:10:04.484865 27929 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260620 06:10:04.485000 27929 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935603120653-11672-0/minicluster-data/ts-2/data,/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935603120653-11672-0/minicluster-data/ts-2/wal
uuid: "5600989c2f634c4cb630839afa98c1a3"
format_stamp: "Formatted at 2026-06-20 06:09:43 on dist-test-slave-9hm5"
server_key: "7c752eff990c445a6e4932608495d08e"
server_key_iv: "0b3b1b24d5500c0211dfa70a90f90c34"
server_key_version: "encryptionkey@0"
I20260620 06:10:04.485529 27929 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935603120653-11672-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935603120653-11672-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935603120653-11672-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:10:04.503315 27929 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:10:04.506570 27929 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:10:04.506776 27929 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:10:04.507356 27929 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 06:10:04.508183 27929 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 06:10:04.508252 27929 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 06:10:04.508324 27929 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 06:10:04.508369 27929 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 06:10:04.517014 27929 rpc_server.cc:307] RPC server started. Bound to: 127.11.102.3:43283
I20260620 06:10:04.517042 28057 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.102.3:43283 every 8 connection(s)
I20260620 06:10:04.517967 27929 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935603120653-11672-0/minicluster-data/ts-2/data/info.pb
I20260620 06:10:04.522727 11672 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9HMp43/build/debug/bin/kudu as pid 27929
Jun 20 06:10:04 dist-test-slave-9hm5 krb5kdc[27019](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935804, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.102.3@KRBTEST.COM for kudu/127.11.102.62@KRBTEST.COM
I20260620 06:10:04.531531 28062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:10:04.523396 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45500 (local address 127.11.102.62:41831)
0620 06:10:04.523646 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0620 06:10:04.523650 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 06:10:04.523779 (+ 129us) server_negotiation.cc:407] Connection header received
0620 06:10:04.523887 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:10:04.523889 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:10:04.523924 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:10:04.523999 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:10:04.524814 (+ 815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:10:04.525575 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:10:04.526627 (+ 1052us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:10:04.526908 (+ 281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:10:04.527011 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:10:04.528389 (+ 1378us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:10:04.528394 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:10:04.528396 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:10:04.528419 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:10:04.529878 (+ 1459us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:10:04.530290 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:10:04.530294 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:10:04.530296 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:10:04.530382 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:10:04.530678 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:10:04.530681 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:10:04.530683 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:10:04.530858 (+ 175us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:10:04.531064 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0620 06:10:04.531211 (+ 147us) server_negotiation.cc:299] Negotiation successful
0620 06:10:04.531342 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":164,"thread_start_us":72,"threads_started":1}
I20260620 06:10:04.533037 28060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:10:04.520099 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.102.62:41831 (local address 127.11.102.3:57245)
0620 06:10:04.520507 (+ 408us) negotiation.cc:107] Waiting for socket to connect
0620 06:10:04.520546 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0620 06:10:04.521667 (+ 1121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:10:04.522166 (+ 499us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:10:04.522172 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:10:04.522608 (+ 436us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:10:04.523277 (+ 669us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:10:04.523300 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:10:04.524429 (+ 1129us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:10:04.524432 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:10:04.525679 (+ 1247us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:10:04.525690 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:10:04.526249 (+ 559us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:10:04.526252 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:10:04.526385 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:10:04.526936 (+ 551us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:10:04.526971 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:10:04.528446 (+ 1475us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:10:04.530238 (+ 1792us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:10:04.530244 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:10:04.530258 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:10:04.530511 (+ 253us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:10:04.530784 (+ 273us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:10:04.530789 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:10:04.530791 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:10:04.530924 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:10:04.531301 (+ 377us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:10:04.531310 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:10:04.531668 (+ 358us) client_negotiation.cc:769] Sending connection context
0620 06:10:04.531893 (+ 225us) client_negotiation.cc:240] Negotiation successful
0620 06:10:04.532222 (+ 329us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":231,"thread_start_us":98,"threads_started":1}
I20260620 06:10:04.533649 27789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:10:04.519727 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.102.3:57245 (local address 127.11.102.62:41831)
0620 06:10:04.519858 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0620 06:10:04.519862 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 06:10:04.520660 (+ 798us) server_negotiation.cc:407] Connection header received
0620 06:10:04.521862 (+ 1202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:10:04.521865 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:10:04.521911 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:10:04.522000 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:10:04.523438 (+ 1438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:10:04.524296 (+ 858us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:10:04.525787 (+ 1491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:10:04.526109 (+ 322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:10:04.526214 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:10:04.528614 (+ 2400us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:10:04.528617 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:10:04.528641 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:10:04.528686 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:10:04.530123 (+ 1437us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:10:04.530613 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:10:04.530618 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:10:04.530619 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:10:04.530696 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:10:04.531025 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:10:04.531029 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:10:04.531030 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:10:04.531200 (+ 170us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:10:04.531325 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0620 06:10:04.533439 (+ 2114us) server_negotiation.cc:299] Negotiation successful
0620 06:10:04.533550 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50,"spinlock_wait_cycles":9088}
I20260620 06:10:04.534408 28058 heartbeater.cc:347] Connected to a master server at 127.11.102.62:41831
I20260620 06:10:04.534623 28058 heartbeater.cc:464] Registering TS with master...
I20260620 06:10:04.535063 28058 heartbeater.cc:511] Master 127.11.102.62:41831 requested a full tablet report, sending...
I20260620 06:10:04.535985 27599 ts_manager.cc:194] Registered new tserver with Master: 5600989c2f634c4cb630839afa98c1a3 (127.11.102.3:43283)
I20260620 06:10:04.536492 27599 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.102.3@KRBTEST.COM'} at 127.11.102.3:57245
I20260620 06:10:04.545166 11672 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 06:10:04.548995 27789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:10:04.545745 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45514 (local address 127.11.102.62:41831)
0620 06:10:04.545858 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0620 06:10:04.545861 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 06:10:04.546139 (+ 278us) server_negotiation.cc:407] Connection header received
0620 06:10:04.546268 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:10:04.546270 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:10:04.546362 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:10:04.546446 (+ 84us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 06:10:04.546659 (+ 213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:10:04.547143 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:10:04.548040 (+ 897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:10:04.548243 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:10:04.548319 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:10:04.548569 (+ 250us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 06:10:04.548698 (+ 129us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0620 06:10:04.548889 (+ 191us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:45514: token expired
Metrics: {"server-negotiator.queue_time_us":42}
W20260620 06:10:04.549075 27789 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:45514: token expired
W20260620 06:10:04.549455 27528 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.11.102.62:41831)
I20260620 06:10:04.556663 27789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:10:04.549936 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45526 (local address 127.11.102.62:41831)
0620 06:10:04.550045 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0620 06:10:04.550048 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 06:10:04.550058 (+ 10us) server_negotiation.cc:407] Connection header received
0620 06:10:04.550085 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:10:04.550087 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:10:04.550118 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:10:04.550178 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:10:04.550759 (+ 581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:10:04.551213 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:10:04.552051 (+ 838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:10:04.552243 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:10:04.552316 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:10:04.553542 (+ 1226us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:10:04.553545 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:10:04.553546 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:10:04.553567 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:10:04.554993 (+ 1426us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:10:04.555480 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:10:04.555482 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:10:04.555484 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:10:04.555558 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:10:04.555837 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:10:04.555839 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:10:04.555841 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:10:04.556003 (+ 162us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:10:04.556102 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0620 06:10:04.556455 (+ 353us) server_negotiation.cc:299] Negotiation successful
0620 06:10:04.556546 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260620 06:10:04.559302 27599 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45526:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 06:10:04.562644 27599 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 06:10:04.570634 27721 tablet_service.cc:1467] Processing CreateTablet for tablet 47cd9415278242fbac0701a0a2e559f3 (DEFAULT_TABLE table=test-table [id=edae2118b9eb4979928c85450f25f5cb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 06:10:04.571308 27721 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 47cd9415278242fbac0701a0a2e559f3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 06:10:04.574419 27679 tablet_bootstrap.cc:492] T 47cd9415278242fbac0701a0a2e559f3 P 3fd44bea82cc4e89bfc9ada83a1fd302: Bootstrap starting.
I20260620 06:10:04.574870 27794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:10:04.569370 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.102.3:43283 (local address 127.0.0.1:49398)
0620 06:10:04.569576 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0620 06:10:04.569592 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 06:10:04.569667 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:10:04.570468 (+ 801us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:10:04.570471 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:10:04.570489 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 06:10:04.570602 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:10:04.570613 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:10:04.572004 (+ 1391us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:10:04.572010 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:10:04.572978 (+ 968us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:10:04.572991 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:10:04.574419 (+ 1428us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:10:04.574422 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:10:04.574524 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:10:04.574575 (+ 51us) client_negotiation.cc:769] Sending connection context
0620 06:10:04.574659 (+ 84us) client_negotiation.cc:240] Negotiation successful
0620 06:10:04.574740 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":112}
I20260620 06:10:04.574946 28067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:10:04.569541 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49398 (local address 127.11.102.3:43283)
0620 06:10:04.569929 (+ 388us) server_negotiation.cc:206] Beginning negotiation
0620 06:10:04.569941 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 06:10:04.569973 (+ 32us) server_negotiation.cc:407] Connection header received
0620 06:10:04.570054 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:10:04.570060 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:10:04.570304 (+ 244us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:10:04.570475 (+ 171us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 06:10:04.570731 (+ 256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:10:04.571882 (+ 1151us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:10:04.573104 (+ 1222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:10:04.574276 (+ 1172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:10:04.574421 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:10:04.574584 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0620 06:10:04.574692 (+ 108us) server_negotiation.cc:299] Negotiation successful
0620 06:10:04.574792 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":290,"thread_start_us":205,"threads_started":1}
I20260620 06:10:04.575314 27679 tablet_bootstrap.cc:654] T 47cd9415278242fbac0701a0a2e559f3 P 3fd44bea82cc4e89bfc9ada83a1fd302: Neither blocks nor log segments found. Creating new log.
I20260620 06:10:04.576100 28068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:10:04.569632 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.102.2:40107 (local address 127.0.0.1:47760)
0620 06:10:04.570414 (+ 782us) negotiation.cc:107] Waiting for socket to connect
0620 06:10:04.570429 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 06:10:04.570525 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:10:04.571280 (+ 755us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:10:04.571283 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:10:04.571301 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 06:10:04.571503 (+ 202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:10:04.571511 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:10:04.573012 (+ 1501us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:10:04.573018 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:10:04.573971 (+ 953us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:10:04.573981 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:10:04.575580 (+ 1599us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:10:04.575585 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:10:04.575747 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:10:04.575783 (+ 36us) client_negotiation.cc:769] Sending connection context
0620 06:10:04.575866 (+ 83us) client_negotiation.cc:240] Negotiation successful
0620 06:10:04.575937 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":719,"thread_start_us":109,"threads_started":1}
I20260620 06:10:04.576548 28069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:10:04.570004 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47760 (local address 127.11.102.2:40107)
0620 06:10:04.570332 (+ 328us) server_negotiation.cc:206] Beginning negotiation
0620 06:10:04.570344 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 06:10:04.570616 (+ 272us) server_negotiation.cc:407] Connection header received
0620 06:10:04.570700 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:10:04.570705 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:10:04.570914 (+ 209us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:10:04.571083 (+ 169us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 06:10:04.571795 (+ 712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:10:04.572875 (+ 1080us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:10:04.574096 (+ 1221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:10:04.575445 (+ 1349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:10:04.575560 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:10:04.575707 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0620 06:10:04.576279 (+ 572us) server_negotiation.cc:299] Negotiation successful
0620 06:10:04.576382 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":229,"thread_start_us":125,"threads_started":1}
I20260620 06:10:04.577258 27679 tablet_bootstrap.cc:492] T 47cd9415278242fbac0701a0a2e559f3 P 3fd44bea82cc4e89bfc9ada83a1fd302: No bootstrap required, opened a new log
I20260620 06:10:04.577353 27679 ts_tablet_manager.cc:1397] T 47cd9415278242fbac0701a0a2e559f3 P 3fd44bea82cc4e89bfc9ada83a1fd302: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260620 06:10:04.577844 27679 raft_consensus.cc:348] T 47cd9415278242fbac0701a0a2e559f3 P 3fd44bea82cc4e89bfc9ada83a1fd302 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5600989c2f634c4cb630839afa98c1a3" member_type: VOTER last_known_addr { host: "127.11.102.3" port: 43283 } } peers { permanent_uuid: "cfae6937c171468db2b68d87b1149dea" member_type: VOTER last_known_addr { host: "127.11.102.2" port: 40107 } } peers { permanent_uuid: "3fd44bea82cc4e89bfc9ada83a1fd302" member_type: VOTER last_known_addr { host: "127.11.102.1" port: 37171 } }
I20260620 06:10:04.577953 27679 raft_consensus.cc:374] T 47cd9415278242fbac0701a0a2e559f3 P 3fd44bea82cc4e89bfc9ada83a1fd302 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 06:10:04.577988 27679 raft_consensus.cc:729] T 47cd9415278242fbac0701a0a2e559f3 P 3fd44bea82cc4e89bfc9ada83a1fd302 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3fd44bea82cc4e89bfc9ada83a1fd302, State: Initialized, Role: FOLLOWER
I20260620 06:10:04.578148 27679 consensus_queue.cc:260] T 47cd9415278242fbac0701a0a2e559f3 P 3fd44bea82cc4e89bfc9ada83a1fd302 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5600989c2f634c4cb630839afa98c1a3" member_type: VOTER last_known_addr { host: "127.11.102.3" port: 43283 } } peers { permanent_uuid: "cfae6937c171468db2b68d87b1149dea" member_type: VOTER last_known_addr { host: "127.11.102.2" port: 40107 } } peers { permanent_uuid: "3fd44bea82cc4e89bfc9ada83a1fd302" member_type: VOTER last_known_addr { host: "127.11.102.1" port: 37171 } }
I20260620 06:10:04.578465 27679 ts_tablet_manager.cc:1428] T 47cd9415278242fbac0701a0a2e559f3 P 3fd44bea82cc4e89bfc9ada83a1fd302: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260620 06:10:04.578146 27992 tablet_service.cc:1467] Processing CreateTablet for tablet 47cd9415278242fbac0701a0a2e559f3 (DEFAULT_TABLE table=test-table [id=edae2118b9eb4979928c85450f25f5cb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 06:10:04.579205 27992 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 47cd9415278242fbac0701a0a2e559f3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 06:10:04.580675 27860 tablet_service.cc:1467] Processing CreateTablet for tablet 47cd9415278242fbac0701a0a2e559f3 (DEFAULT_TABLE table=test-table [id=edae2118b9eb4979928c85450f25f5cb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 06:10:04.581624 27860 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 47cd9415278242fbac0701a0a2e559f3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 06:10:04.585564 27818 tablet_bootstrap.cc:492] T 47cd9415278242fbac0701a0a2e559f3 P cfae6937c171468db2b68d87b1149dea: Bootstrap starting.
I20260620 06:10:04.586540 28071 tablet_bootstrap.cc:492] T 47cd9415278242fbac0701a0a2e559f3 P 5600989c2f634c4cb630839afa98c1a3: Bootstrap starting.
I20260620 06:10:04.587885 27818 tablet_bootstrap.cc:654] T 47cd9415278242fbac0701a0a2e559f3 P cfae6937c171468db2b68d87b1149dea: Neither blocks nor log segments found. Creating new log.
I20260620 06:10:04.588733 28071 tablet_bootstrap.cc:654] T 47cd9415278242fbac0701a0a2e559f3 P 5600989c2f634c4cb630839afa98c1a3: Neither blocks nor log segments found. Creating new log.
I20260620 06:10:04.588822 27818 log.cc:824] T 47cd9415278242fbac0701a0a2e559f3 P cfae6937c171468db2b68d87b1149dea: Log is configured to *not* fsync() on all Append() calls
I20260620 06:10:04.589895 28071 log.cc:824] T 47cd9415278242fbac0701a0a2e559f3 P 5600989c2f634c4cb630839afa98c1a3: Log is configured to *not* fsync() on all Append() calls
I20260620 06:10:04.591040 27818 tablet_bootstrap.cc:492] T 47cd9415278242fbac0701a0a2e559f3 P cfae6937c171468db2b68d87b1149dea: No bootstrap required, opened a new log
I20260620 06:10:04.591221 27818 ts_tablet_manager.cc:1397] T 47cd9415278242fbac0701a0a2e559f3 P cfae6937c171468db2b68d87b1149dea: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260620 06:10:04.591915 28071 tablet_bootstrap.cc:492] T 47cd9415278242fbac0701a0a2e559f3 P 5600989c2f634c4cb630839afa98c1a3: No bootstrap required, opened a new log
I20260620 06:10:04.592106 28071 ts_tablet_manager.cc:1397] T 47cd9415278242fbac0701a0a2e559f3 P 5600989c2f634c4cb630839afa98c1a3: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260620 06:10:04.595275 27818 raft_consensus.cc:348] T 47cd9415278242fbac0701a0a2e559f3 P cfae6937c171468db2b68d87b1149dea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5600989c2f634c4cb630839afa98c1a3" member_type: VOTER last_known_addr { host: "127.11.102.3" port: 43283 } } peers { permanent_uuid: "cfae6937c171468db2b68d87b1149dea" member_type: VOTER last_known_addr { host: "127.11.102.2" port: 40107 } } peers { permanent_uuid: "3fd44bea82cc4e89bfc9ada83a1fd302" member_type: VOTER last_known_addr { host: "127.11.102.1" port: 37171 } }
I20260620 06:10:04.595570 27818 raft_consensus.cc:374] T 47cd9415278242fbac0701a0a2e559f3 P cfae6937c171468db2b68d87b1149dea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 06:10:04.595655 27818 raft_consensus.cc:729] T 47cd9415278242fbac0701a0a2e559f3 P cfae6937c171468db2b68d87b1149dea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cfae6937c171468db2b68d87b1149dea, State: Initialized, Role: FOLLOWER
I20260620 06:10:04.595866 28071 raft_consensus.cc:348] T 47cd9415278242fbac0701a0a2e559f3 P 5600989c2f634c4cb630839afa98c1a3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5600989c2f634c4cb630839afa98c1a3" member_type: VOTER last_known_addr { host: "127.11.102.3" port: 43283 } } peers { permanent_uuid: "cfae6937c171468db2b68d87b1149dea" member_type: VOTER last_known_addr { host: "127.11.102.2" port: 40107 } } peers { permanent_uuid: "3fd44bea82cc4e89bfc9ada83a1fd302" member_type: VOTER last_known_addr { host: "127.11.102.1" port: 37171 } }
I20260620 06:10:04.596150 28071 raft_consensus.cc:374] T 47cd9415278242fbac0701a0a2e559f3 P 5600989c2f634c4cb630839afa98c1a3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 06:10:04.596122 27818 consensus_queue.cc:260] T 47cd9415278242fbac0701a0a2e559f3 P cfae6937c171468db2b68d87b1149dea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5600989c2f634c4cb630839afa98c1a3" member_type: VOTER last_known_addr { host: "127.11.102.3" port: 43283 } } peers { permanent_uuid: "cfae6937c171468db2b68d87b1149dea" member_type: VOTER last_known_addr { host: "127.11.102.2" port: 40107 } } peers { permanent_uuid: "3fd44bea82cc4e89bfc9ada83a1fd302" member_type: VOTER last_known_addr { host: "127.11.102.1" port: 37171 } }
I20260620 06:10:04.596204 28071 raft_consensus.cc:729] T 47cd9415278242fbac0701a0a2e559f3 P 5600989c2f634c4cb630839afa98c1a3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5600989c2f634c4cb630839afa98c1a3, State: Initialized, Role: FOLLOWER
I20260620 06:10:04.596638 28071 consensus_queue.cc:260] T 47cd9415278242fbac0701a0a2e559f3 P 5600989c2f634c4cb630839afa98c1a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5600989c2f634c4cb630839afa98c1a3" member_type: VOTER last_known_addr { host: "127.11.102.3" port: 43283 } } peers { permanent_uuid: "cfae6937c171468db2b68d87b1149dea" member_type: VOTER last_known_addr { host: "127.11.102.2" port: 40107 } } peers { permanent_uuid: "3fd44bea82cc4e89bfc9ada83a1fd302" member_type: VOTER last_known_addr { host: "127.11.102.1" port: 37171 } }
I20260620 06:10:04.596804 27926 heartbeater.cc:503] Master 127.11.102.62:41831 was elected leader, sending a full tablet report...
I20260620 06:10:04.597185 27818 ts_tablet_manager.cc:1428] T 47cd9415278242fbac0701a0a2e559f3 P cfae6937c171468db2b68d87b1149dea: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260620 06:10:04.597249 28058 heartbeater.cc:503] Master 127.11.102.62:41831 was elected leader, sending a full tablet report...
I20260620 06:10:04.598515 28071 ts_tablet_manager.cc:1428] T 47cd9415278242fbac0701a0a2e559f3 P 5600989c2f634c4cb630839afa98c1a3: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
W20260620 06:10:04.610653 27788 tablet.cc:2369] T 47cd9415278242fbac0701a0a2e559f3 P 3fd44bea82cc4e89bfc9ada83a1fd302: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 06:10:04.610772 27793 raft_consensus.cc:482] T 47cd9415278242fbac0701a0a2e559f3 P 3fd44bea82cc4e89bfc9ada83a1fd302 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 06:10:04.611017 27793 raft_consensus.cc:504] T 47cd9415278242fbac0701a0a2e559f3 P 3fd44bea82cc4e89bfc9ada83a1fd302 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5600989c2f634c4cb630839afa98c1a3" member_type: VOTER last_known_addr { host: "127.11.102.3" port: 43283 } } peers { permanent_uuid: "cfae6937c171468db2b68d87b1149dea" member_type: VOTER last_known_addr { host: "127.11.102.2" port: 40107 } } peers { permanent_uuid: "3fd44bea82cc4e89bfc9ada83a1fd302" member_type: VOTER last_known_addr { host: "127.11.102.1" port: 37171 } }
I20260620 06:10:04.612217 27793 leader_election.cc:288] T 47cd9415278242fbac0701a0a2e559f3 P 3fd44bea82cc4e89bfc9ada83a1fd302 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5600989c2f634c4cb630839afa98c1a3 (127.11.102.3:43283), cfae6937c171468db2b68d87b1149dea (127.11.102.2:40107)
I20260620 06:10:04.617491 28067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:10:04.612580 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.102.1:34391 (local address 127.11.102.3:43283)
0620 06:10:04.612718 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0620 06:10:04.612724 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 06:10:04.612743 (+ 19us) server_negotiation.cc:407] Connection header received
0620 06:10:04.612871 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:10:04.612877 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:10:04.612965 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:10:04.613056 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 06:10:04.613394 (+ 338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:10:04.614252 (+ 858us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:10:04.616215 (+ 1963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:10:04.616882 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:10:04.616996 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:10:04.617039 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0620 06:10:04.617281 (+ 242us) server_negotiation.cc:299] Negotiation successful
0620 06:10:04.617354 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260620 06:10:04.617491 28069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:10:04.612595 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.102.1:43535 (local address 127.11.102.2:40107)
0620 06:10:04.612783 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0620 06:10:04.612794 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 06:10:04.612865 (+ 71us) server_negotiation.cc:407] Connection header received
0620 06:10:04.613003 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:10:04.613010 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:10:04.613068 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:10:04.613142 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 06:10:04.613393 (+ 251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:10:04.614262 (+ 869us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:10:04.616128 (+ 1866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:10:04.616803 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:10:04.616975 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:10:04.617029 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0620 06:10:04.617284 (+ 255us) server_negotiation.cc:299] Negotiation successful
0620 06:10:04.617354 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":99}
I20260620 06:10:04.617534 27790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:10:04.612411 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.102.3:43283 (local address 127.11.102.1:34391)
0620 06:10:04.612614 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0620 06:10:04.612631 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 06:10:04.612728 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:10:04.613078 (+ 350us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:10:04.613080 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:10:04.613093 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 06:10:04.613241 (+ 148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:10:04.613252 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:10:04.614388 (+ 1136us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:10:04.614392 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:10:04.615999 (+ 1607us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:10:04.616011 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:10:04.617004 (+ 993us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:10:04.617007 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:10:04.617099 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:10:04.617160 (+ 61us) client_negotiation.cc:769] Sending connection context
0620 06:10:04.617247 (+ 87us) client_negotiation.cc:240] Negotiation successful
0620 06:10:04.617373 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"mutex_wait_us":199}
I20260620 06:10:04.617753 28076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:10:04.612442 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.102.2:40107 (local address 127.11.102.1:43535)
0620 06:10:04.612799 (+ 357us) negotiation.cc:107] Waiting for socket to connect
0620 06:10:04.612814 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 06:10:04.612902 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:10:04.613158 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:10:04.613160 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:10:04.613171 (+ 11us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 06:10:04.613283 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:10:04.613292 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:10:04.614388 (+ 1096us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:10:04.614392 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:10:04.615999 (+ 1607us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:10:04.616011 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:10:04.616950 (+ 939us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:10:04.616956 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:10:04.617078 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:10:04.617187 (+ 109us) client_negotiation.cc:769] Sending connection context
0620 06:10:04.617260 (+ 73us) client_negotiation.cc:240] Negotiation successful
0620 06:10:04.617373 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":289,"mutex_wait_us":195,"thread_start_us":86,"threads_started":1}
I20260620 06:10:04.618268 28012 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47cd9415278242fbac0701a0a2e559f3" candidate_uuid: "3fd44bea82cc4e89bfc9ada83a1fd302" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5600989c2f634c4cb630839afa98c1a3" is_pre_election: true
I20260620 06:10:04.618468 27880 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47cd9415278242fbac0701a0a2e559f3" candidate_uuid: "3fd44bea82cc4e89bfc9ada83a1fd302" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cfae6937c171468db2b68d87b1149dea" is_pre_election: true
I20260620 06:10:04.618575 28012 raft_consensus.cc:2454] T 47cd9415278242fbac0701a0a2e559f3 P 5600989c2f634c4cb630839afa98c1a3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3fd44bea82cc4e89bfc9ada83a1fd302 in term 0.
I20260620 06:10:04.618780 27880 raft_consensus.cc:2454] T 47cd9415278242fbac0701a0a2e559f3 P cfae6937c171468db2b68d87b1149dea [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3fd44bea82cc4e89bfc9ada83a1fd302 in term 0.
I20260620 06:10:04.619088 27674 leader_election.cc:302] T 47cd9415278242fbac0701a0a2e559f3 P 3fd44bea82cc4e89bfc9ada83a1fd302 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3fd44bea82cc4e89bfc9ada83a1fd302, 5600989c2f634c4cb630839afa98c1a3; no voters:
I20260620 06:10:04.619397 27793 raft_consensus.cc:2781] T 47cd9415278242fbac0701a0a2e559f3 P 3fd44bea82cc4e89bfc9ada83a1fd302 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 06:10:04.619474 27793 raft_consensus.cc:482] T 47cd9415278242fbac0701a0a2e559f3 P 3fd44bea82cc4e89bfc9ada83a1fd302 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 06:10:04.619505 27793 raft_consensus.cc:3037] T 47cd9415278242fbac0701a0a2e559f3 P 3fd44bea82cc4e89bfc9ada83a1fd302 [term 0 FOLLOWER]: Advancing to term 1
I20260620 06:10:04.620590 27793 raft_consensus.cc:504] T 47cd9415278242fbac0701a0a2e559f3 P 3fd44bea82cc4e89bfc9ada83a1fd302 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5600989c2f634c4cb630839afa98c1a3" member_type: VOTER last_known_addr { host: "127.11.102.3" port: 43283 } } peers { permanent_uuid: "cfae6937c171468db2b68d87b1149dea" member_type: VOTER last_known_addr { host: "127.11.102.2" port: 40107 } } peers { permanent_uuid: "3fd44bea82cc4e89bfc9ada83a1fd302" member_type: VOTER last_known_addr { host: "127.11.102.1" port: 37171 } }
I20260620 06:10:04.620951 27793 leader_election.cc:288] T 47cd9415278242fbac0701a0a2e559f3 P 3fd44bea82cc4e89bfc9ada83a1fd302 [CANDIDATE]: Term 1 election: Requested vote from peers 5600989c2f634c4cb630839afa98c1a3 (127.11.102.3:43283), cfae6937c171468db2b68d87b1149dea (127.11.102.2:40107)
I20260620 06:10:04.621361 27880 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47cd9415278242fbac0701a0a2e559f3" candidate_uuid: "3fd44bea82cc4e89bfc9ada83a1fd302" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cfae6937c171468db2b68d87b1149dea"
I20260620 06:10:04.621390 28012 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47cd9415278242fbac0701a0a2e559f3" candidate_uuid: "3fd44bea82cc4e89bfc9ada83a1fd302" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5600989c2f634c4cb630839afa98c1a3"
I20260620 06:10:04.621469 27880 raft_consensus.cc:3037] T 47cd9415278242fbac0701a0a2e559f3 P cfae6937c171468db2b68d87b1149dea [term 0 FOLLOWER]: Advancing to term 1
I20260620 06:10:04.621519 28012 raft_consensus.cc:3037] T 47cd9415278242fbac0701a0a2e559f3 P 5600989c2f634c4cb630839afa98c1a3 [term 0 FOLLOWER]: Advancing to term 1
I20260620 06:10:04.622488 28012 raft_consensus.cc:2454] T 47cd9415278242fbac0701a0a2e559f3 P 5600989c2f634c4cb630839afa98c1a3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3fd44bea82cc4e89bfc9ada83a1fd302 in term 1.
I20260620 06:10:04.622493 27880 raft_consensus.cc:2454] T 47cd9415278242fbac0701a0a2e559f3 P cfae6937c171468db2b68d87b1149dea [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3fd44bea82cc4e89bfc9ada83a1fd302 in term 1.
I20260620 06:10:04.622782 27674 leader_election.cc:302] T 47cd9415278242fbac0701a0a2e559f3 P 3fd44bea82cc4e89bfc9ada83a1fd302 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3fd44bea82cc4e89bfc9ada83a1fd302, 5600989c2f634c4cb630839afa98c1a3; no voters:
I20260620 06:10:04.622970 27793 raft_consensus.cc:2781] T 47cd9415278242fbac0701a0a2e559f3 P 3fd44bea82cc4e89bfc9ada83a1fd302 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 06:10:04.623205 27793 raft_consensus.cc:686] T 47cd9415278242fbac0701a0a2e559f3 P 3fd44bea82cc4e89bfc9ada83a1fd302 [term 1 LEADER]: Becoming Leader. State: Replica: 3fd44bea82cc4e89bfc9ada83a1fd302, State: Running, Role: LEADER
I20260620 06:10:04.623488 27793 consensus_queue.cc:237] T 47cd9415278242fbac0701a0a2e559f3 P 3fd44bea82cc4e89bfc9ada83a1fd302 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5600989c2f634c4cb630839afa98c1a3" member_type: VOTER last_known_addr { host: "127.11.102.3" port: 43283 } } peers { permanent_uuid: "cfae6937c171468db2b68d87b1149dea" member_type: VOTER last_known_addr { host: "127.11.102.2" port: 40107 } } peers { permanent_uuid: "3fd44bea82cc4e89bfc9ada83a1fd302" member_type: VOTER last_known_addr { host: "127.11.102.1" port: 37171 } }
I20260620 06:10:04.626519 27599 catalog_manager.cc:5495] T 47cd9415278242fbac0701a0a2e559f3 P 3fd44bea82cc4e89bfc9ada83a1fd302 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3fd44bea82cc4e89bfc9ada83a1fd302 (127.11.102.1). New cstate: current_term: 1 leader_uuid: "3fd44bea82cc4e89bfc9ada83a1fd302" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5600989c2f634c4cb630839afa98c1a3" member_type: VOTER last_known_addr { host: "127.11.102.3" port: 43283 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cfae6937c171468db2b68d87b1149dea" member_type: VOTER last_known_addr { host: "127.11.102.2" port: 40107 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3fd44bea82cc4e89bfc9ada83a1fd302" member_type: VOTER last_known_addr { host: "127.11.102.1" port: 37171 } health_report { overall_health: HEALTHY } } }
I20260620 06:10:04.647773 27795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:10:04.644259 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53428 (local address 127.11.102.1:37171)
0620 06:10:04.644408 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0620 06:10:04.644415 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 06:10:04.644442 (+ 27us) server_negotiation.cc:407] Connection header received
0620 06:10:04.644535 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:10:04.644537 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:10:04.644588 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:10:04.644715 (+ 127us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 06:10:04.644968 (+ 253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:10:04.645505 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:10:04.646326 (+ 821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:10:04.646569 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:10:04.646679 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:10:04.646949 (+ 270us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 06:10:04.647366 (+ 417us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 06:10:04.647502 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0620 06:10:04.647550 (+ 48us) server_negotiation.cc:299] Negotiation successful
0620 06:10:04.647604 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260620 06:10:04.651859 27880 raft_consensus.cc:1261] T 47cd9415278242fbac0701a0a2e559f3 P cfae6937c171468db2b68d87b1149dea [term 1 FOLLOWER]: Refusing update from remote peer 3fd44bea82cc4e89bfc9ada83a1fd302: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 06:10:04.651860 28012 raft_consensus.cc:1261] T 47cd9415278242fbac0701a0a2e559f3 P 5600989c2f634c4cb630839afa98c1a3 [term 1 FOLLOWER]: Refusing update from remote peer 3fd44bea82cc4e89bfc9ada83a1fd302: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 06:10:04.652595 28077 consensus_queue.cc:1035] T 47cd9415278242fbac0701a0a2e559f3 P 3fd44bea82cc4e89bfc9ada83a1fd302 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cfae6937c171468db2b68d87b1149dea" member_type: VOTER last_known_addr { host: "127.11.102.2" port: 40107 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 06:10:04.652781 27793 consensus_queue.cc:1035] T 47cd9415278242fbac0701a0a2e559f3 P 3fd44bea82cc4e89bfc9ada83a1fd302 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5600989c2f634c4cb630839afa98c1a3" member_type: VOTER last_known_addr { host: "127.11.102.3" port: 43283 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 06:10:04.659842 28080 mvcc.cc:205] Tried to move back new op lower bound from 7298809055846567936 to 7298809055740620800. Current Snapshot: MvccSnapshot[applied={T|T < 7298809055846567936}]
I20260620 06:10:04.668192 27599 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45526:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 06:10:04.668354 27599 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45526:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 06:10:04.670652 27599 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 08cf14c4a270431082d354a7a84ab2fd: Sending DeleteTablet for 3 replicas of tablet 47cd9415278242fbac0701a0a2e559f3
I20260620 06:10:04.671330 27992 tablet_service.cc:1514] Processing DeleteTablet for tablet 47cd9415278242fbac0701a0a2e559f3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 06:10:04 UTC) from {username='kudu', principal='kudu/127.11.102.62@KRBTEST.COM'} at 127.0.0.1:49398
I20260620 06:10:04.671530 27860 tablet_service.cc:1514] Processing DeleteTablet for tablet 47cd9415278242fbac0701a0a2e559f3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 06:10:04 UTC) from {username='kudu', principal='kudu/127.11.102.62@KRBTEST.COM'} at 127.0.0.1:47760
I20260620 06:10:04.671901 28089 tablet_replica.cc:330] T 47cd9415278242fbac0701a0a2e559f3 P 5600989c2f634c4cb630839afa98c1a3: stopping tablet replica
I20260620 06:10:04.672035 28090 tablet_replica.cc:330] T 47cd9415278242fbac0701a0a2e559f3 P cfae6937c171468db2b68d87b1149dea: stopping tablet replica
I20260620 06:10:04.672005 27721 tablet_service.cc:1514] Processing DeleteTablet for tablet 47cd9415278242fbac0701a0a2e559f3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 06:10:04 UTC) from {username='kudu', principal='kudu/127.11.102.62@KRBTEST.COM'} at 127.0.0.1:53424
I20260620 06:10:04.672341 28089 raft_consensus.cc:2229] T 47cd9415278242fbac0701a0a2e559f3 P 5600989c2f634c4cb630839afa98c1a3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 06:10:04.672389 28090 raft_consensus.cc:2229] T 47cd9415278242fbac0701a0a2e559f3 P cfae6937c171468db2b68d87b1149dea [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 06:10:04.672441 11672 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9HMp43/build/debug/bin/kudu with pid 27637
I20260620 06:10:04.672729 28090 raft_consensus.cc:2258] T 47cd9415278242fbac0701a0a2e559f3 P cfae6937c171468db2b68d87b1149dea [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 06:10:04.673274 28089 raft_consensus.cc:2258] T 47cd9415278242fbac0701a0a2e559f3 P 5600989c2f634c4cb630839afa98c1a3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 06:10:04.674423 28089 ts_tablet_manager.cc:1905] T 47cd9415278242fbac0701a0a2e559f3 P 5600989c2f634c4cb630839afa98c1a3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 06:10:04.674420 28090 ts_tablet_manager.cc:1905] T 47cd9415278242fbac0701a0a2e559f3 P cfae6937c171468db2b68d87b1149dea: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 06:10:04.676577 28089 ts_tablet_manager.cc:1918] T 47cd9415278242fbac0701a0a2e559f3 P 5600989c2f634c4cb630839afa98c1a3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 06:10:04.676702 28089 log.cc:1196] T 47cd9415278242fbac0701a0a2e559f3 P 5600989c2f634c4cb630839afa98c1a3: Deleting WAL directory at /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935603120653-11672-0/minicluster-data/ts-2/wal/wals/47cd9415278242fbac0701a0a2e559f3
I20260620 06:10:04.677023 28089 ts_tablet_manager.cc:1939] T 47cd9415278242fbac0701a0a2e559f3 P 5600989c2f634c4cb630839afa98c1a3: Deleting consensus metadata
I20260620 06:10:04.677882 28090 ts_tablet_manager.cc:1918] T 47cd9415278242fbac0701a0a2e559f3 P cfae6937c171468db2b68d87b1149dea: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 06:10:04.677970 28090 log.cc:1196] T 47cd9415278242fbac0701a0a2e559f3 P cfae6937c171468db2b68d87b1149dea: Deleting WAL directory at /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935603120653-11672-0/minicluster-data/ts-1/wal/wals/47cd9415278242fbac0701a0a2e559f3
I20260620 06:10:04.678172 27584 catalog_manager.cc:4841] TS 5600989c2f634c4cb630839afa98c1a3 (127.11.102.3:43283): tablet 47cd9415278242fbac0701a0a2e559f3 (table test-table [id=edae2118b9eb4979928c85450f25f5cb]) successfully deleted
I20260620 06:10:04.678269 28090 ts_tablet_manager.cc:1939] T 47cd9415278242fbac0701a0a2e559f3 P cfae6937c171468db2b68d87b1149dea: Deleting consensus metadata
I20260620 06:10:04.679396 27585 catalog_manager.cc:4841] TS cfae6937c171468db2b68d87b1149dea (127.11.102.2:40107): tablet 47cd9415278242fbac0701a0a2e559f3 (table test-table [id=edae2118b9eb4979928c85450f25f5cb]) successfully deleted
W20260620 06:10:04.681453 27585 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.11.102.1:37171 (error 108)
I20260620 06:10:04.681525 11672 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9HMp43/build/debug/bin/kudu with pid 27792
I20260620 06:10:04.688024 11672 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9HMp43/build/debug/bin/kudu with pid 27929
I20260620 06:10:04.688577 28068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:10:04.688169 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.102.1:37171 (local address 127.0.0.1:53430)
0620 06:10:04.688376 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0620 06:10:04.688450 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.102.1:37171: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101}
W20260620 06:10:04.688812 27585 catalog_manager.cc:4584] TS 3fd44bea82cc4e89bfc9ada83a1fd302 (127.11.102.1:37171): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 47cd9415278242fbac0701a0a2e559f3: Network error: Client connection negotiation failed: client connection to 127.11.102.1:37171: connect: Connection refused (error 111)
I20260620 06:10:04.693967 11672 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9HMp43/build/debug/bin/kudu with pid 27566
I20260620 06:10:24.700537 11672 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9HMp43/build/debug/bin/kudu
/tmp/dist-test-task9HMp43/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935603120653-11672-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935603120653-11672-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935603120653-11672-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935603120653-11672-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.102.62:41831
--webserver_interface=127.11.102.62
--webserver_port=41901
--keytab_file=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935603120653-11672-0/krb5kdc/kudu_127.11.102.62.keytab
--principal=kudu/127.11.102.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.102.20:38439
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.102.62:41831
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935603120653-11672-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935603120653-11672-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935603120653-11672-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 06:10:24.812834 28093 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:10:24.813148 28093 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:10:24.813189 28093 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:10:24.817394 28093 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260620 06:10:24.817449 28093 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260620 06:10:24.817466 28093 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260620 06:10:24.817479 28093 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 06:10:24.817497 28093 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:10:24.817554 28093 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:10:24.817577 28093 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 06:10:24.817607 28093 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 06:10:24.823032 28093 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.102.20:38439
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935603120653-11672-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935603120653-11672-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.11.102.62:41831
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935603120653-11672-0/krb5kdc/kudu_127.11.102.62.keytab
--principal=kudu/127.11.102.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.102.62:41831
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935603120653-11672-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.102.62
--webserver_port=41901
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28093
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935603120653-11672-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 05:28:48 UTC on ccycloud.quasar-fixhms.root.comops.site
build id 4457020
I20260620 06:10:24.824225 28093 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:10:24.825290 28093 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 06:10:24.834474 28093 server_base.cc:1034] running on GCE node
W20260620 06:10:24.834440 28099 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:10:24.834417 28101 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:10:24.834430 28098 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:10:24.835404 28093 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:10:24.836655 28093 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:10:24.837855 28093 hybrid_clock.cc:648] HybridClock initialized: now 1781935824837839 us; error 30 us; skew 500 ppm
Jun 20 06:10:24 dist-test-slave-9hm5 krb5kdc[27019](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935824, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.102.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 06:10:24.840818 28093 init.cc:377] Logged in from keytab as kudu/127.11.102.62@KRBTEST.COM (short username kudu)
I20260620 06:10:24.841735 28093 webserver.cc:460] Webserver started at http://127.11.102.62:41901/ using document root <none> and password file <none>
I20260620 06:10:24.842319 28093 fs_manager.cc:359] Metadata directory not provided
I20260620 06:10:24.842414 28093 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:10:24.846405 28093 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.001s sys 0.001s
I20260620 06:10:24.849602 28108 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:10:24.850795 28093 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260620 06:10:24.850937 28093 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935603120653-11672-0/minicluster-data/master-0/data,/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935603120653-11672-0/minicluster-data/master-0/wal
uuid: "08cf14c4a270431082d354a7a84ab2fd"
format_stamp: "Formatted at 2026-06-20 06:09:42 on dist-test-slave-9hm5"
server_key: "30abf25c0b3089745fb551a08427a3b0"
server_key_iv: "5d2f4a95c2affcce55c4ce2dcb4b6639"
server_key_version: "encryptionkey@0"
I20260620 06:10:24.851495 28093 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935603120653-11672-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935603120653-11672-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935603120653-11672-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:10:24.890478 28093 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:10:24.893472 28093 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:10:24.893687 28093 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:10:24.900619 28093 rpc_server.cc:307] RPC server started. Bound to: 127.11.102.62:41831
I20260620 06:10:24.900916 28160 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.102.62:41831 every 8 connection(s)
I20260620 06:10:24.901706 28093 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935603120653-11672-0/minicluster-data/master-0/data/info.pb
I20260620 06:10:24.905845 28161 sys_catalog.cc:263] Verifying existing consensus state
I20260620 06:10:24.907502 28161 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 08cf14c4a270431082d354a7a84ab2fd: Bootstrap starting.
I20260620 06:10:24.908016 11672 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9HMp43/build/debug/bin/kudu as pid 28093
I20260620 06:10:24.908455 11672 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9HMp43/build/debug/bin/kudu
/tmp/dist-test-task9HMp43/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935603120653-11672-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935603120653-11672-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935603120653-11672-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935603120653-11672-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.102.1:37171
--local_ip_for_outbound_sockets=127.11.102.1
--tserver_master_addrs=127.11.102.62:41831
--webserver_port=34149
--webserver_interface=127.11.102.1
--keytab_file=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935603120653-11672-0/krb5kdc/kudu_127.11.102.1.keytab
--principal=kudu/127.11.102.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.102.20:38439
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935603120653-11672-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935603120653-11672-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935603120653-11672-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 06:10:24.931674 28161 log.cc:824] T 00000000000000000000000000000000 P 08cf14c4a270431082d354a7a84ab2fd: Log is configured to *not* fsync() on all Append() calls
I20260620 06:10:24.946105 28161 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 08cf14c4a270431082d354a7a84ab2fd: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 06:10:24.946557 28161 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 08cf14c4a270431082d354a7a84ab2fd: Bootstrap complete.
I20260620 06:10:24.952019 28161 raft_consensus.cc:348] T 00000000000000000000000000000000 P 08cf14c4a270431082d354a7a84ab2fd [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "08cf14c4a270431082d354a7a84ab2fd" member_type: VOTER last_known_addr { host: "127.11.102.62" port: 41831 } }
I20260620 06:10:24.952546 28161 raft_consensus.cc:729] T 00000000000000000000000000000000 P 08cf14c4a270431082d354a7a84ab2fd [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 08cf14c4a270431082d354a7a84ab2fd, State: Initialized, Role: FOLLOWER
I20260620 06:10:24.953084 28161 consensus_queue.cc:260] T 00000000000000000000000000000000 P 08cf14c4a270431082d354a7a84ab2fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "08cf14c4a270431082d354a7a84ab2fd" member_type: VOTER last_known_addr { host: "127.11.102.62" port: 41831 } }
I20260620 06:10:24.953238 28161 raft_consensus.cc:388] T 00000000000000000000000000000000 P 08cf14c4a270431082d354a7a84ab2fd [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 06:10:24.953311 28161 raft_consensus.cc:482] T 00000000000000000000000000000000 P 08cf14c4a270431082d354a7a84ab2fd [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 06:10:24.953406 28161 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 08cf14c4a270431082d354a7a84ab2fd [term 2 FOLLOWER]: Advancing to term 3
I20260620 06:10:24.955895 28161 raft_consensus.cc:504] T 00000000000000000000000000000000 P 08cf14c4a270431082d354a7a84ab2fd [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "08cf14c4a270431082d354a7a84ab2fd" member_type: VOTER last_known_addr { host: "127.11.102.62" port: 41831 } }
I20260620 06:10:24.956283 28161 leader_election.cc:302] T 00000000000000000000000000000000 P 08cf14c4a270431082d354a7a84ab2fd [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 08cf14c4a270431082d354a7a84ab2fd; no voters:
I20260620 06:10:24.956626 28161 leader_election.cc:288] T 00000000000000000000000000000000 P 08cf14c4a270431082d354a7a84ab2fd [CANDIDATE]: Term 3 election: Requested vote from peers
I20260620 06:10:24.956796 28165 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 08cf14c4a270431082d354a7a84ab2fd [term 3 FOLLOWER]: Leader election won for term 3
I20260620 06:10:24.957072 28165 raft_consensus.cc:686] T 00000000000000000000000000000000 P 08cf14c4a270431082d354a7a84ab2fd [term 3 LEADER]: Becoming Leader. State: Replica: 08cf14c4a270431082d354a7a84ab2fd, State: Running, Role: LEADER
I20260620 06:10:24.957355 28165 consensus_queue.cc:237] T 00000000000000000000000000000000 P 08cf14c4a270431082d354a7a84ab2fd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "08cf14c4a270431082d354a7a84ab2fd" member_type: VOTER last_known_addr { host: "127.11.102.62" port: 41831 } }
I20260620 06:10:24.957965 28161 sys_catalog.cc:564] T 00000000000000000000000000000000 P 08cf14c4a270431082d354a7a84ab2fd [sys.catalog]: configured and running, proceeding with master startup.
I20260620 06:10:24.958251 28165 sys_catalog.cc:455] T 00000000000000000000000000000000 P 08cf14c4a270431082d354a7a84ab2fd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 08cf14c4a270431082d354a7a84ab2fd. Latest consensus state: current_term: 3 leader_uuid: "08cf14c4a270431082d354a7a84ab2fd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "08cf14c4a270431082d354a7a84ab2fd" member_type: VOTER last_known_addr { host: "127.11.102.62" port: 41831 } } }
I20260620 06:10:24.958376 28165 sys_catalog.cc:458] T 00000000000000000000000000000000 P 08cf14c4a270431082d354a7a84ab2fd [sys.catalog]: This master's current role is: LEADER
I20260620 06:10:24.958647 28166 sys_catalog.cc:455] T 00000000000000000000000000000000 P 08cf14c4a270431082d354a7a84ab2fd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "08cf14c4a270431082d354a7a84ab2fd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "08cf14c4a270431082d354a7a84ab2fd" member_type: VOTER last_known_addr { host: "127.11.102.62" port: 41831 } } }
I20260620 06:10:24.958724 28166 sys_catalog.cc:458] T 00000000000000000000000000000000 P 08cf14c4a270431082d354a7a84ab2fd [sys.catalog]: This master's current role is: LEADER
I20260620 06:10:24.958786 28169 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 06:10:24.963824 28169 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 06:10:24.964584 28169 catalog_manager.cc:1259] Loaded cluster ID: c5bf0008db3641f5aa12f375b47b3a94
I20260620 06:10:24.964663 28169 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 06:10:24.966387 28169 catalog_manager.cc:1504] Loading token signing keys...
I20260620 06:10:24.966992 28169 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 08cf14c4a270431082d354a7a84ab2fd: Loaded TSK: 0
I20260620 06:10:24.967084 28169 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 08cf14c4a270431082d354a7a84ab2fd: Loaded TSK: 1
F20260620 06:10:24.967857 28169 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7fde6ae299ad google::LogMessage::Fail() at ??:0
@ 0x7fde6ae2c88a google::LogMessage::SendToLog() at ??:0
@ 0x7fde6ae294de google::LogMessage::Flush() at ??:0
@ 0x7fde6ae2aa19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7fde6daed18d kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7fde6dae5f0d kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7fde7578e7bb kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7fde7576899c kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7fde75769178 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlvE3_clEv at ??:0
@ 0x7fde757aadf8 _ZNSt17_Function_handlerIFN4kudu6StatusEvEZNS0_6master14CatalogManager24PrepareForLeadershipTaskEvEUlvE3_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x7fde757c982b std::function<>::operator()() at ??:0
@ 0x7fde75768b4f _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_ at ??:0
@ 0x7fde75769f3e kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7fde75766579 _ZZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvENKUlvE_clEv at ??:0
@ 0x7fde757aa640 _ZNSt17_Function_handlerIFvvEZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7fde6d07b3d7 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7fde6d07bc83 _ZZN4kudu10ThreadPool12CreateThreadEvENKUlvE_clEv at ??:0
@ 0x7fde6d07d3f2 _ZNSt17_Function_handlerIFvvEZN4kudu10ThreadPool12CreateThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7fde6d06c512 kudu::Thread::SuperviseThread() at ??:0
@ 0x7fde6e6b06db start_thread at ??:0
@ 0x7fde6992761f clone at ??:0
W20260620 06:10:25.076473 28163 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:10:25.076740 28163 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:10:25.076786 28163 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:10:25.081498 28163 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:10:25.081569 28163 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:10:25.081655 28163 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.102.1
I20260620 06:10:25.087219 28163 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.102.20:38439
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935603120653-11672-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935603120653-11672-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935603120653-11672-0/krb5kdc/kudu_127.11.102.1.keytab
--principal=kudu/127.11.102.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.102.1:37171
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935603120653-11672-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.11.102.1
--webserver_port=34149
--tserver_master_addrs=127.11.102.62:41831
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28163
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.102.1
--log_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935603120653-11672-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 05:28:48 UTC on ccycloud.quasar-fixhms.root.comops.site
build id 4457020
I20260620 06:10:25.088433 28163 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:10:25.089493 28163 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 06:10:25.098961 28211 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:10:25.098956 28209 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:10:25.098934 28208 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:10:25.100111 28163 server_base.cc:1034] running on GCE node
I20260620 06:10:25.100541 28163 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:10:25.101233 28163 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:10:25.102419 28163 hybrid_clock.cc:648] HybridClock initialized: now 1781935825102391 us; error 46 us; skew 500 ppm
Jun 20 06:10:25 dist-test-slave-9hm5 krb5kdc[27019](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935825, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.102.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 06:10:25.105320 28163 init.cc:377] Logged in from keytab as kudu/127.11.102.1@KRBTEST.COM (short username kudu)
I20260620 06:10:25.106261 28163 webserver.cc:460] Webserver started at http://127.11.102.1:34149/ using document root <none> and password file <none>
I20260620 06:10:25.106863 28163 fs_manager.cc:359] Metadata directory not provided
I20260620 06:10:25.106957 28163 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:10:25.111007 28163 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.001s
I20260620 06:10:25.113472 28218 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:10:25.114835 28163 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260620 06:10:25.114974 28163 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935603120653-11672-0/minicluster-data/ts-0/data,/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935603120653-11672-0/minicluster-data/ts-0/wal
uuid: "3fd44bea82cc4e89bfc9ada83a1fd302"
format_stamp: "Formatted at 2026-06-20 06:09:42 on dist-test-slave-9hm5"
server_key: "87f03c04be86da2aa07d685cba61f524"
server_key_iv: "0050b961ddf0b580701336a7fd81ae80"
server_key_version: "encryptionkey@0"
I20260620 06:10:25.115548 28163 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935603120653-11672-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935603120653-11672-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935603120653-11672-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:10:25.149927 28163 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:10:25.153224 28163 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:10:25.153434 28163 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:10:25.154112 28163 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 06:10:25.155294 28225 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260620 06:10:25.160786 28163 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260620 06:10:25.160893 28163 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.006s user 0.000s sys 0.000s
I20260620 06:10:25.160981 28163 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260620 06:10:25.162658 28163 ts_tablet_manager.cc:610] Registered 1 tablets
I20260620 06:10:25.162730 28163 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260620 06:10:25.163026 28225 tablet_bootstrap.cc:492] T 47cd9415278242fbac0701a0a2e559f3 P 3fd44bea82cc4e89bfc9ada83a1fd302: Bootstrap starting.
I20260620 06:10:25.173430 28163 rpc_server.cc:307] RPC server started. Bound to: 127.11.102.1:37171
I20260620 06:10:25.173676 28332 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.102.1:37171 every 8 connection(s)
I20260620 06:10:25.174474 28163 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935603120653-11672-0/minicluster-data/ts-0/data/info.pb
I20260620 06:10:25.176854 11672 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9HMp43/build/debug/bin/kudu as pid 28163
I20260620 06:10:25.177338 11672 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9HMp43/build/debug/bin/kudu
/tmp/dist-test-task9HMp43/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935603120653-11672-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935603120653-11672-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935603120653-11672-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935603120653-11672-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.102.2:40107
--local_ip_for_outbound_sockets=127.11.102.2
--tserver_master_addrs=127.11.102.62:41831
--webserver_port=42091
--webserver_interface=127.11.102.2
--keytab_file=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935603120653-11672-0/krb5kdc/kudu_127.11.102.2.keytab
--principal=kudu/127.11.102.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.102.20:38439
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935603120653-11672-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935603120653-11672-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935603120653-11672-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 06:10:25.185045 28335 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 06:10:25.176504 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.102.62:41831 (local address 127.11.102.1:37625)
0620 06:10:25.178037 (+ 1533us) negotiation.cc:107] Waiting for socket to connect
0620 06:10:25.178113 (+ 76us) client_negotiation.cc:174] Beginning negotiation
0620 06:10:25.179341 (+ 1228us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:10:25.184205 (+ 4864us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.102.62:41831: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":1282,"thread_start_us":159,"threads_started":1}
W20260620 06:10:25.185601 28333 heartbeater.cc:644] Failed to heartbeat to 127.11.102.62:41831 (0 consecutive failures): Network error: Failed to ping master at 127.11.102.62:41831: Client connection negotiation failed: client connection to 127.11.102.62:41831: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260620 06:10:25.186259 28335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:10:25.186020 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.102.62:41831 (local address 127.11.102.1:49865)
0620 06:10:25.186138 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0620 06:10:25.186206 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.102.62:41831: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260620 06:10:25.186450 28225 log.cc:824] T 47cd9415278242fbac0701a0a2e559f3 P 3fd44bea82cc4e89bfc9ada83a1fd302: Log is configured to *not* fsync() on all Append() calls
I20260620 06:10:25.187004 28335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:10:25.186809 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.102.62:41831 (local address 127.11.102.1:50453)
0620 06:10:25.186905 (+ 96us) negotiation.cc:107] Waiting for socket to connect
0620 06:10:25.186950 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.102.62:41831: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260620 06:10:25.187314 28333 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260620 06:10:25.193532 28225 tablet_bootstrap.cc:492] T 47cd9415278242fbac0701a0a2e559f3 P 3fd44bea82cc4e89bfc9ada83a1fd302: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 06:10:25.193920 28225 tablet_bootstrap.cc:492] T 47cd9415278242fbac0701a0a2e559f3 P 3fd44bea82cc4e89bfc9ada83a1fd302: Bootstrap complete.
I20260620 06:10:25.195055 28225 ts_tablet_manager.cc:1397] T 47cd9415278242fbac0701a0a2e559f3 P 3fd44bea82cc4e89bfc9ada83a1fd302: Time spent bootstrapping tablet: real 0.032s user 0.020s sys 0.007s
I20260620 06:10:25.197464 28225 raft_consensus.cc:348] T 47cd9415278242fbac0701a0a2e559f3 P 3fd44bea82cc4e89bfc9ada83a1fd302 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5600989c2f634c4cb630839afa98c1a3" member_type: VOTER last_known_addr { host: "127.11.102.3" port: 43283 } } peers { permanent_uuid: "cfae6937c171468db2b68d87b1149dea" member_type: VOTER last_known_addr { host: "127.11.102.2" port: 40107 } } peers { permanent_uuid: "3fd44bea82cc4e89bfc9ada83a1fd302" member_type: VOTER last_known_addr { host: "127.11.102.1" port: 37171 } }
I20260620 06:10:25.197901 28225 raft_consensus.cc:729] T 47cd9415278242fbac0701a0a2e559f3 P 3fd44bea82cc4e89bfc9ada83a1fd302 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3fd44bea82cc4e89bfc9ada83a1fd302, State: Initialized, Role: FOLLOWER
I20260620 06:10:25.198403 28225 consensus_queue.cc:260] T 47cd9415278242fbac0701a0a2e559f3 P 3fd44bea82cc4e89bfc9ada83a1fd302 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5600989c2f634c4cb630839afa98c1a3" member_type: VOTER last_known_addr { host: "127.11.102.3" port: 43283 } } peers { permanent_uuid: "cfae6937c171468db2b68d87b1149dea" member_type: VOTER last_known_addr { host: "127.11.102.2" port: 40107 } } peers { permanent_uuid: "3fd44bea82cc4e89bfc9ada83a1fd302" member_type: VOTER last_known_addr { host: "127.11.102.1" port: 37171 } }
I20260620 06:10:25.199311 28225 ts_tablet_manager.cc:1428] T 47cd9415278242fbac0701a0a2e559f3 P 3fd44bea82cc4e89bfc9ada83a1fd302: Time spent starting tablet: real 0.004s user 0.006s sys 0.001s
I20260620 06:10:25.199740 28335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:10:25.199405 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.102.62:41831 (local address 127.11.102.1:45401)
0620 06:10:25.199534 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0620 06:10:25.199657 (+ 123us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.102.62:41831: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20260620 06:10:25.200358 28335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:10:25.200153 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.102.62:41831 (local address 127.11.102.1:36517)
0620 06:10:25.200256 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0620 06:10:25.200301 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.102.62:41831: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260620 06:10:25.290297 28336 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:10:25.290592 28336 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:10:25.290674 28336 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:10:25.294668 28336 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:10:25.294763 28336 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:10:25.294857 28336 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.102.2
I20260620 06:10:25.300088 28336 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.102.20:38439
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935603120653-11672-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935603120653-11672-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935603120653-11672-0/krb5kdc/kudu_127.11.102.2.keytab
--principal=kudu/127.11.102.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.102.2:40107
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935603120653-11672-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.11.102.2
--webserver_port=42091
--tserver_master_addrs=127.11.102.62:41831
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28336
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.102.2
--log_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935603120653-11672-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 05:28:48 UTC on ccycloud.quasar-fixhms.root.comops.site
build id 4457020
I20260620 06:10:25.301189 28336 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:10:25.302227 28336 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 06:10:25.312098 28343 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:10:25.312197 28336 server_base.cc:1034] running on GCE node
W20260620 06:10:25.312100 28346 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:10:25.312150 28344 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:10:25.312825 28336 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:10:25.313536 28336 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:10:25.314692 28336 hybrid_clock.cc:648] HybridClock initialized: now 1781935825314680 us; error 53 us; skew 500 ppm
Jun 20 06:10:25 dist-test-slave-9hm5 krb5kdc[27019](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935825, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.102.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 06:10:25.317516 28336 init.cc:377] Logged in from keytab as kudu/127.11.102.2@KRBTEST.COM (short username kudu)
I20260620 06:10:25.318439 28336 webserver.cc:460] Webserver started at http://127.11.102.2:42091/ using document root <none> and password file <none>
I20260620 06:10:25.319129 28336 fs_manager.cc:359] Metadata directory not provided
I20260620 06:10:25.319222 28336 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:10:25.323207 28336 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260620 06:10:25.325709 28353 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:10:25.326977 28336 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 06:10:25.327121 28336 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935603120653-11672-0/minicluster-data/ts-1/data,/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935603120653-11672-0/minicluster-data/ts-1/wal
uuid: "cfae6937c171468db2b68d87b1149dea"
format_stamp: "Formatted at 2026-06-20 06:09:43 on dist-test-slave-9hm5"
server_key: "4e9bccdc31a644917c436f65b57a1180"
server_key_iv: "bf1fa8eb3d43a242ea056778541df575"
server_key_version: "encryptionkey@0"
I20260620 06:10:25.327733 28336 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935603120653-11672-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935603120653-11672-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935603120653-11672-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:10:25.356782 28336 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:10:25.360139 28336 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:10:25.360355 28336 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:10:25.360975 28336 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 06:10:25.361835 28336 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 06:10:25.361907 28336 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 06:10:25.361980 28336 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 06:10:25.362031 28336 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 06:10:25.371050 28336 rpc_server.cc:307] RPC server started. Bound to: 127.11.102.2:40107
I20260620 06:10:25.371074 28466 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.102.2:40107 every 8 connection(s)
I20260620 06:10:25.372134 28336 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935603120653-11672-0/minicluster-data/ts-1/data/info.pb
I20260620 06:10:25.375577 11672 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9HMp43/build/debug/bin/kudu as pid 28336
I20260620 06:10:25.376087 11672 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9HMp43/build/debug/bin/kudu
/tmp/dist-test-task9HMp43/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935603120653-11672-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935603120653-11672-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935603120653-11672-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935603120653-11672-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.102.3:43283
--local_ip_for_outbound_sockets=127.11.102.3
--tserver_master_addrs=127.11.102.62:41831
--webserver_port=42325
--webserver_interface=127.11.102.3
--keytab_file=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935603120653-11672-0/krb5kdc/kudu_127.11.102.3.keytab
--principal=kudu/127.11.102.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.102.20:38439
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935603120653-11672-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935603120653-11672-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935603120653-11672-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 06:10:25.376230 28469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:10:25.374383 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.102.62:41831 (local address 127.11.102.2:37465)
0620 06:10:25.374875 (+ 492us) negotiation.cc:107] Waiting for socket to connect
0620 06:10:25.375328 (+ 453us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.102.62:41831: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":291,"thread_start_us":108,"threads_started":1}
W20260620 06:10:25.376706 28467 heartbeater.cc:644] Failed to heartbeat to 127.11.102.62:41831 (0 consecutive failures): Network error: Failed to ping master at 127.11.102.62:41831: Client connection negotiation failed: client connection to 127.11.102.62:41831: connect: Connection refused (error 111)
I20260620 06:10:25.377384 28469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:10:25.377169 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.102.62:41831 (local address 127.11.102.2:54975)
0620 06:10:25.377283 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0620 06:10:25.377332 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.102.62:41831: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260620 06:10:25.378058 28469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:10:25.377841 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.102.62:41831 (local address 127.11.102.2:56001)
0620 06:10:25.377955 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0620 06:10:25.378004 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.102.62:41831: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260620 06:10:25.378222 28467 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260620 06:10:25.487589 28470 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:10:25.487918 28470 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:10:25.487963 28470 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:10:25.491979 28470 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:10:25.492061 28470 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:10:25.492177 28470 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.102.3
I20260620 06:10:25.497385 28470 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.102.20:38439
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935603120653-11672-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935603120653-11672-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935603120653-11672-0/krb5kdc/kudu_127.11.102.3.keytab
--principal=kudu/127.11.102.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.102.3:43283
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935603120653-11672-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.11.102.3
--webserver_port=42325
--tserver_master_addrs=127.11.102.62:41831
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28470
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.102.3
--log_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935603120653-11672-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 05:28:48 UTC on ccycloud.quasar-fixhms.root.comops.site
build id 4457020
I20260620 06:10:25.498409 28470 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:10:25.499394 28470 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 06:10:25.508838 28475 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:10:25.508975 28470 server_base.cc:1034] running on GCE node
W20260620 06:10:25.508868 28478 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:10:25.509020 28476 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:10:25.509496 28470 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:10:25.510113 28470 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:10:25.511274 28470 hybrid_clock.cc:648] HybridClock initialized: now 1781935825511260 us; error 26 us; skew 500 ppm
Jun 20 06:10:25 dist-test-slave-9hm5 krb5kdc[27019](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935825, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.102.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 06:10:25.514087 28470 init.cc:377] Logged in from keytab as kudu/127.11.102.3@KRBTEST.COM (short username kudu)
I20260620 06:10:25.514886 28470 webserver.cc:460] Webserver started at http://127.11.102.3:42325/ using document root <none> and password file <none>
I20260620 06:10:25.515413 28470 fs_manager.cc:359] Metadata directory not provided
I20260620 06:10:25.515472 28470 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:10:25.519245 28470 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.001s
I20260620 06:10:25.521310 28485 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:10:25.522436 28470 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260620 06:10:25.522552 28470 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935603120653-11672-0/minicluster-data/ts-2/data,/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935603120653-11672-0/minicluster-data/ts-2/wal
uuid: "5600989c2f634c4cb630839afa98c1a3"
format_stamp: "Formatted at 2026-06-20 06:09:43 on dist-test-slave-9hm5"
server_key: "7c752eff990c445a6e4932608495d08e"
server_key_iv: "0b3b1b24d5500c0211dfa70a90f90c34"
server_key_version: "encryptionkey@0"
I20260620 06:10:25.523079 28470 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935603120653-11672-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935603120653-11672-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935603120653-11672-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:10:25.538728 28470 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:10:25.542208 28470 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:10:25.542412 28470 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:10:25.543032 28470 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 06:10:25.543918 28470 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 06:10:25.543967 28470 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 06:10:25.544035 28470 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 06:10:25.544075 28470 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 06:10:25.553109 28470 rpc_server.cc:307] RPC server started. Bound to: 127.11.102.3:43283
I20260620 06:10:25.553134 28598 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.102.3:43283 every 8 connection(s)
I20260620 06:10:25.554157 28470 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935603120653-11672-0/minicluster-data/ts-2/data/info.pb
I20260620 06:10:25.558018 28601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:10:25.556365 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.102.62:41831 (local address 127.11.102.3:34203)
0620 06:10:25.556908 (+ 543us) negotiation.cc:107] Waiting for socket to connect
0620 06:10:25.557323 (+ 415us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.102.62:41831: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":343,"thread_start_us":118,"threads_started":1}
W20260620 06:10:25.558493 28599 heartbeater.cc:644] Failed to heartbeat to 127.11.102.62:41831 (0 consecutive failures): Network error: Failed to ping master at 127.11.102.62:41831: Client connection negotiation failed: client connection to 127.11.102.62:41831: connect: Connection refused (error 111)
I20260620 06:10:25.559041 28601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:10:25.558844 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.102.62:41831 (local address 127.11.102.3:48077)
0620 06:10:25.558942 (+ 98us) negotiation.cc:107] Waiting for socket to connect
0620 06:10:25.558989 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.102.62:41831: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
I20260620 06:10:25.559697 28601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:10:25.559452 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.102.62:41831 (local address 127.11.102.3:39059)
0620 06:10:25.559549 (+ 97us) negotiation.cc:107] Waiting for socket to connect
0620 06:10:25.559589 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.102.62:41831: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260620 06:10:25.559899 28599 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260620 06:10:25.563160 11672 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9HMp43/build/debug/bin/kudu as pid 28470
I20260620 06:10:26.201695 28603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:10:26.201120 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.102.62:41831 (local address 127.11.102.1:37049)
0620 06:10:26.201444 (+ 324us) negotiation.cc:107] Waiting for socket to connect
0620 06:10:26.201513 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.102.62:41831: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":234,"thread_start_us":83,"threads_started":1}
I20260620 06:10:26.379287 28604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:10:26.378836 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.102.62:41831 (local address 127.11.102.2:52265)
0620 06:10:26.379102 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0620 06:10:26.379163 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.102.62:41831: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":84,"threads_started":1}
I20260620 06:10:26.560811 28605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:10:26.560392 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.102.62:41831 (local address 127.11.102.3:50451)
0620 06:10:26.560641 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0620 06:10:26.560717 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.102.62:41831: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":103,"threads_started":1}
I20260620 06:10:26.607971 28606 raft_consensus.cc:482] T 47cd9415278242fbac0701a0a2e559f3 P 3fd44bea82cc4e89bfc9ada83a1fd302 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 06:10:26.608239 28606 raft_consensus.cc:504] T 47cd9415278242fbac0701a0a2e559f3 P 3fd44bea82cc4e89bfc9ada83a1fd302 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5600989c2f634c4cb630839afa98c1a3" member_type: VOTER last_known_addr { host: "127.11.102.3" port: 43283 } } peers { permanent_uuid: "cfae6937c171468db2b68d87b1149dea" member_type: VOTER last_known_addr { host: "127.11.102.2" port: 40107 } } peers { permanent_uuid: "3fd44bea82cc4e89bfc9ada83a1fd302" member_type: VOTER last_known_addr { host: "127.11.102.1" port: 37171 } }
I20260620 06:10:26.609372 28606 leader_election.cc:288] T 47cd9415278242fbac0701a0a2e559f3 P 3fd44bea82cc4e89bfc9ada83a1fd302 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5600989c2f634c4cb630839afa98c1a3 (127.11.102.3:43283), cfae6937c171468db2b68d87b1149dea (127.11.102.2:40107)
Jun 20 06:10:26 dist-test-slave-9hm5 krb5kdc[27019](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935825, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.102.1@KRBTEST.COM for kudu/127.11.102.3@KRBTEST.COM
Jun 20 06:10:26 dist-test-slave-9hm5 krb5kdc[27019](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935825, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.102.1@KRBTEST.COM for kudu/127.11.102.2@KRBTEST.COM
I20260620 06:10:26.627166 28603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:10:26.609634 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.102.2:40107 (local address 127.11.102.1:42141)
0620 06:10:26.609846 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0620 06:10:26.609864 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 06:10:26.610006 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:10:26.611795 (+ 1789us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:10:26.611802 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:10:26.612359 (+ 557us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:10:26.612848 (+ 489us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:10:26.612863 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:10:26.614327 (+ 1464us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:10:26.614332 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:10:26.615395 (+ 1063us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:10:26.615407 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:10:26.616234 (+ 827us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:10:26.616238 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:10:26.616489 (+ 251us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:10:26.620574 (+ 4085us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:10:26.620588 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:10:26.622647 (+ 2059us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:10:26.624870 (+ 2223us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:10:26.624872 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:10:26.624874 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:10:26.625089 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:10:26.625472 (+ 383us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:10:26.625475 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:10:26.625476 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:10:26.625601 (+ 125us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:10:26.626361 (+ 760us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:10:26.626364 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:10:26.626662 (+ 298us) client_negotiation.cc:769] Sending connection context
0620 06:10:26.626802 (+ 140us) client_negotiation.cc:240] Negotiation successful
0620 06:10:26.627036 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120,"mutex_wait_us":213,"spinlock_wait_cycles":12032}
I20260620 06:10:26.627431 28607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:10:26.609634 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.102.3:43283 (local address 127.11.102.1:35015)
0620 06:10:26.610003 (+ 369us) negotiation.cc:107] Waiting for socket to connect
0620 06:10:26.610018 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 06:10:26.610119 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:10:26.611796 (+ 1677us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:10:26.611803 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:10:26.612349 (+ 546us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:10:26.612848 (+ 499us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:10:26.612863 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:10:26.614293 (+ 1430us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:10:26.614296 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:10:26.615400 (+ 1104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:10:26.615410 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:10:26.616270 (+ 860us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:10:26.616274 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:10:26.616501 (+ 227us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:10:26.620552 (+ 4051us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:10:26.620575 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:10:26.622290 (+ 1715us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:10:26.624651 (+ 2361us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:10:26.624660 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:10:26.624677 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:10:26.625084 (+ 407us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:10:26.625441 (+ 357us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:10:26.625446 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:10:26.625450 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:10:26.625612 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:10:26.626342 (+ 730us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:10:26.626348 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:10:26.626665 (+ 317us) client_negotiation.cc:769] Sending connection context
0620 06:10:26.626831 (+ 166us) client_negotiation.cc:240] Negotiation successful
0620 06:10:26.627126 (+ 295us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":304,"mutex_wait_us":217,"spinlock_wait_cycles":19584,"thread_start_us":88,"threads_started":1}
I20260620 06:10:26.627441 28608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:10:26.609860 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.102.1:35015 (local address 127.11.102.3:43283)
0620 06:10:26.610280 (+ 420us) server_negotiation.cc:206] Beginning negotiation
0620 06:10:26.610305 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0620 06:10:26.610344 (+ 39us) server_negotiation.cc:407] Connection header received
0620 06:10:26.611012 (+ 668us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:10:26.611017 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:10:26.611372 (+ 355us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:10:26.611732 (+ 360us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:10:26.613041 (+ 1309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:10:26.614119 (+ 1078us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:10:26.615674 (+ 1555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:10:26.616139 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:10:26.616318 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:10:26.622595 (+ 6277us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:10:26.622600 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:10:26.622614 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:10:26.622645 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:10:26.624521 (+ 1876us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:10:26.625206 (+ 685us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:10:26.625212 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:10:26.625217 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:10:26.625342 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:10:26.625738 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:10:26.625741 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:10:26.625743 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:10:26.626222 (+ 479us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:10:26.626368 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0620 06:10:26.626962 (+ 594us) server_negotiation.cc:299] Negotiation successful
0620 06:10:26.627255 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":298,"thread_start_us":69,"threads_started":1}
I20260620 06:10:26.628104 28609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:10:26.609857 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.102.1:42141 (local address 127.11.102.2:40107)
0620 06:10:26.610297 (+ 440us) server_negotiation.cc:206] Beginning negotiation
0620 06:10:26.610317 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0620 06:10:26.610348 (+ 31us) server_negotiation.cc:407] Connection header received
0620 06:10:26.611011 (+ 663us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:10:26.611017 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:10:26.611371 (+ 354us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:10:26.611722 (+ 351us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:10:26.613042 (+ 1320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:10:26.614194 (+ 1152us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:10:26.615604 (+ 1410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:10:26.616102 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:10:26.616286 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:10:26.622851 (+ 6565us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:10:26.622859 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:10:26.622876 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:10:26.622913 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:10:26.624726 (+ 1813us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:10:26.625223 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:10:26.625228 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:10:26.625232 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:10:26.625347 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:10:26.625718 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:10:26.625722 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:10:26.625724 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:10:26.626236 (+ 512us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:10:26.627431 (+ 1195us) server_negotiation.cc:1035] Waiting for connection context
0620 06:10:26.627564 (+ 133us) server_negotiation.cc:299] Negotiation successful
0620 06:10:26.627877 (+ 313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":326,"thread_start_us":121,"threads_started":1}
W20260620 06:10:26.630165 28220 leader_election.cc:341] T 47cd9415278242fbac0701a0a2e559f3 P 3fd44bea82cc4e89bfc9ada83a1fd302 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 5600989c2f634c4cb630839afa98c1a3 (127.11.102.3:43283): Not found: Tablet not found: 47cd9415278242fbac0701a0a2e559f3
W20260620 06:10:26.630342 28221 leader_election.cc:341] T 47cd9415278242fbac0701a0a2e559f3 P 3fd44bea82cc4e89bfc9ada83a1fd302 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer cfae6937c171468db2b68d87b1149dea (127.11.102.2:40107): Not found: Tablet not found: 47cd9415278242fbac0701a0a2e559f3
I20260620 06:10:26.630425 28221 leader_election.cc:302] T 47cd9415278242fbac0701a0a2e559f3 P 3fd44bea82cc4e89bfc9ada83a1fd302 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3fd44bea82cc4e89bfc9ada83a1fd302; no voters: 5600989c2f634c4cb630839afa98c1a3, cfae6937c171468db2b68d87b1149dea
I20260620 06:10:26.630693 28606 raft_consensus.cc:2726] T 47cd9415278242fbac0701a0a2e559f3 P 3fd44bea82cc4e89bfc9ada83a1fd302 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 06:10:27.202901 28610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:10:27.202483 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.102.62:41831 (local address 127.11.102.1:53099)
0620 06:10:27.202745 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0620 06:10:27.202809 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.102.62:41831: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":79,"threads_started":1}
I20260620 06:10:27.380551 28611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:10:27.380131 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.102.62:41831 (local address 127.11.102.2:53819)
0620 06:10:27.380400 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0620 06:10:27.380459 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.102.62:41831: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":81,"threads_started":1}
I20260620 06:10:27.561950 28612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:10:27.561512 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.102.62:41831 (local address 127.11.102.3:55487)
0620 06:10:27.561782 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0620 06:10:27.561843 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.102.62:41831: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":92,"threads_started":1}
I20260620 06:10:28.204378 28613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:10:28.203870 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.102.62:41831 (local address 127.11.102.1:36527)
0620 06:10:28.204203 (+ 333us) negotiation.cc:107] Waiting for socket to connect
0620 06:10:28.204279 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.102.62:41831: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":254,"thread_start_us":79,"threads_started":1}
I20260620 06:10:28.381908 28614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:10:28.381428 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.102.62:41831 (local address 127.11.102.2:50493)
0620 06:10:28.381725 (+ 297us) negotiation.cc:107] Waiting for socket to connect
0620 06:10:28.381797 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.102.62:41831: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":211,"thread_start_us":118,"threads_started":1}
I20260620 06:10:28.563340 28615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:10:28.562851 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.102.62:41831 (local address 127.11.102.3:59245)
0620 06:10:28.563166 (+ 315us) negotiation.cc:107] Waiting for socket to connect
0620 06:10:28.563232 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.102.62:41831: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":234,"thread_start_us":96,"threads_started":1}
I20260620 06:10:28.579078 28616 raft_consensus.cc:482] T 47cd9415278242fbac0701a0a2e559f3 P 3fd44bea82cc4e89bfc9ada83a1fd302 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 06:10:28.579267 28616 raft_consensus.cc:504] T 47cd9415278242fbac0701a0a2e559f3 P 3fd44bea82cc4e89bfc9ada83a1fd302 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5600989c2f634c4cb630839afa98c1a3" member_type: VOTER last_known_addr { host: "127.11.102.3" port: 43283 } } peers { permanent_uuid: "cfae6937c171468db2b68d87b1149dea" member_type: VOTER last_known_addr { host: "127.11.102.2" port: 40107 } } peers { permanent_uuid: "3fd44bea82cc4e89bfc9ada83a1fd302" member_type: VOTER last_known_addr { host: "127.11.102.1" port: 37171 } }
I20260620 06:10:28.579774 28616 leader_election.cc:288] T 47cd9415278242fbac0701a0a2e559f3 P 3fd44bea82cc4e89bfc9ada83a1fd302 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5600989c2f634c4cb630839afa98c1a3 (127.11.102.3:43283), cfae6937c171468db2b68d87b1149dea (127.11.102.2:40107)
W20260620 06:10:28.580646 28221 leader_election.cc:341] T 47cd9415278242fbac0701a0a2e559f3 P 3fd44bea82cc4e89bfc9ada83a1fd302 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer cfae6937c171468db2b68d87b1149dea (127.11.102.2:40107): Not found: Tablet not found: 47cd9415278242fbac0701a0a2e559f3
W20260620 06:10:28.580741 28220 leader_election.cc:341] T 47cd9415278242fbac0701a0a2e559f3 P 3fd44bea82cc4e89bfc9ada83a1fd302 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 5600989c2f634c4cb630839afa98c1a3 (127.11.102.3:43283): Not found: Tablet not found: 47cd9415278242fbac0701a0a2e559f3
I20260620 06:10:28.580862 28220 leader_election.cc:302] T 47cd9415278242fbac0701a0a2e559f3 P 3fd44bea82cc4e89bfc9ada83a1fd302 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3fd44bea82cc4e89bfc9ada83a1fd302; no voters: 5600989c2f634c4cb630839afa98c1a3, cfae6937c171468db2b68d87b1149dea
I20260620 06:10:28.581055 28616 raft_consensus.cc:2726] T 47cd9415278242fbac0701a0a2e559f3 P 3fd44bea82cc4e89bfc9ada83a1fd302 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 06:10:29.205713 28617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:10:29.205221 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.102.62:41831 (local address 127.11.102.1:35807)
0620 06:10:29.205556 (+ 335us) negotiation.cc:107] Waiting for socket to connect
0620 06:10:29.205617 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.102.62:41831: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":254,"thread_start_us":99,"threads_started":1}
I20260620 06:10:29.383381 28618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:10:29.382822 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.102.62:41831 (local address 127.11.102.2:55431)
0620 06:10:29.383190 (+ 368us) negotiation.cc:107] Waiting for socket to connect
0620 06:10:29.383258 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.102.62:41831: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":276,"thread_start_us":147,"threads_started":1}
I20260620 06:10:29.564621 28619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:10:29.564163 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.102.62:41831 (local address 127.11.102.3:41417)
0620 06:10:29.564465 (+ 302us) negotiation.cc:107] Waiting for socket to connect
0620 06:10:29.564529 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.102.62:41831: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":223,"thread_start_us":76,"threads_started":1}
I20260620 06:10:30.207301 28620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:10:30.206809 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.102.62:41831 (local address 127.11.102.1:55533)
0620 06:10:30.207124 (+ 315us) negotiation.cc:107] Waiting for socket to connect
0620 06:10:30.207187 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.102.62:41831: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":238,"thread_start_us":109,"threads_started":1}
I20260620 06:10:30.384589 28621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:10:30.384186 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.102.62:41831 (local address 127.11.102.2:34685)
0620 06:10:30.384449 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0620 06:10:30.384501 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.102.62:41831: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":97,"threads_started":1}
I20260620 06:10:30.565829 28622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:10:30.565448 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.102.62:41831 (local address 127.11.102.3:55981)
0620 06:10:30.565688 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0620 06:10:30.565739 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.102.62:41831: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":86,"threads_started":1}
I20260620 06:10:31.208518 28623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:10:31.208112 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.102.62:41831 (local address 127.11.102.1:51011)
0620 06:10:31.208366 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0620 06:10:31.208424 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.102.62:41831: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":76,"threads_started":1}
I20260620 06:10:31.385795 28624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:10:31.385397 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.102.62:41831 (local address 127.11.102.2:59275)
0620 06:10:31.385643 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0620 06:10:31.385702 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.102.62:41831: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":74,"threads_started":1}
I20260620 06:10:31.566972 28625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:10:31.566581 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.102.62:41831 (local address 127.11.102.3:36259)
0620 06:10:31.566825 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0620 06:10:31.566885 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.102.62:41831: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":74,"threads_started":1}
I20260620 06:10:31.691919 28626 raft_consensus.cc:482] T 47cd9415278242fbac0701a0a2e559f3 P 3fd44bea82cc4e89bfc9ada83a1fd302 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 06:10:31.692055 28626 raft_consensus.cc:504] T 47cd9415278242fbac0701a0a2e559f3 P 3fd44bea82cc4e89bfc9ada83a1fd302 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5600989c2f634c4cb630839afa98c1a3" member_type: VOTER last_known_addr { host: "127.11.102.3" port: 43283 } } peers { permanent_uuid: "cfae6937c171468db2b68d87b1149dea" member_type: VOTER last_known_addr { host: "127.11.102.2" port: 40107 } } peers { permanent_uuid: "3fd44bea82cc4e89bfc9ada83a1fd302" member_type: VOTER last_known_addr { host: "127.11.102.1" port: 37171 } }
I20260620 06:10:31.692413 28626 leader_election.cc:288] T 47cd9415278242fbac0701a0a2e559f3 P 3fd44bea82cc4e89bfc9ada83a1fd302 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5600989c2f634c4cb630839afa98c1a3 (127.11.102.3:43283), cfae6937c171468db2b68d87b1149dea (127.11.102.2:40107)
W20260620 06:10:31.693162 28220 leader_election.cc:341] T 47cd9415278242fbac0701a0a2e559f3 P 3fd44bea82cc4e89bfc9ada83a1fd302 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 5600989c2f634c4cb630839afa98c1a3 (127.11.102.3:43283): Not found: Tablet not found: 47cd9415278242fbac0701a0a2e559f3
W20260620 06:10:31.693259 28221 leader_election.cc:341] T 47cd9415278242fbac0701a0a2e559f3 P 3fd44bea82cc4e89bfc9ada83a1fd302 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer cfae6937c171468db2b68d87b1149dea (127.11.102.2:40107): Not found: Tablet not found: 47cd9415278242fbac0701a0a2e559f3
I20260620 06:10:31.693351 28221 leader_election.cc:302] T 47cd9415278242fbac0701a0a2e559f3 P 3fd44bea82cc4e89bfc9ada83a1fd302 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3fd44bea82cc4e89bfc9ada83a1fd302; no voters: 5600989c2f634c4cb630839afa98c1a3, cfae6937c171468db2b68d87b1149dea
I20260620 06:10:31.693480 28626 raft_consensus.cc:2726] T 47cd9415278242fbac0701a0a2e559f3 P 3fd44bea82cc4e89bfc9ada83a1fd302 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 06:10:32.209766 28627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:10:32.209296 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.102.62:41831 (local address 127.11.102.1:47791)
0620 06:10:32.209610 (+ 314us) negotiation.cc:107] Waiting for socket to connect
0620 06:10:32.209671 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.102.62:41831: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":235,"thread_start_us":91,"threads_started":1}
I20260620 06:10:32.386950 28628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:10:32.386512 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.102.62:41831 (local address 127.11.102.2:54489)
0620 06:10:32.386805 (+ 293us) negotiation.cc:107] Waiting for socket to connect
0620 06:10:32.386860 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.102.62:41831: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":226,"thread_start_us":89,"threads_started":1}
I20260620 06:10:32.568312 28629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:10:32.567777 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.102.62:41831 (local address 127.11.102.3:42285)
0620 06:10:32.568135 (+ 358us) negotiation.cc:107] Waiting for socket to connect
0620 06:10:32.568210 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.102.62:41831: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":275,"thread_start_us":82,"threads_started":1}
I20260620 06:10:33.210974 28630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:10:33.210548 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.102.62:41831 (local address 127.11.102.1:36971)
0620 06:10:33.210826 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0620 06:10:33.210882 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.102.62:41831: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":206,"thread_start_us":88,"threads_started":1}
I20260620 06:10:33.388170 28631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:10:33.387766 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.102.62:41831 (local address 127.11.102.2:35307)
0620 06:10:33.388003 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0620 06:10:33.388075 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.102.62:41831: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":168,"thread_start_us":71,"threads_started":1}
I20260620 06:10:33.569557 28632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:10:33.569117 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.102.62:41831 (local address 127.11.102.3:42129)
0620 06:10:33.569395 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0620 06:10:33.569463 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.102.62:41831: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":206,"thread_start_us":93,"threads_started":1}
I20260620 06:10:34.212253 28633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:10:34.211814 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.102.62:41831 (local address 127.11.102.1:48415)
0620 06:10:34.212109 (+ 295us) negotiation.cc:107] Waiting for socket to connect
0620 06:10:34.212160 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.102.62:41831: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":226,"thread_start_us":98,"threads_started":1}
I20260620 06:10:34.371186 28634 raft_consensus.cc:482] T 47cd9415278242fbac0701a0a2e559f3 P 3fd44bea82cc4e89bfc9ada83a1fd302 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 06:10:34.371320 28634 raft_consensus.cc:504] T 47cd9415278242fbac0701a0a2e559f3 P 3fd44bea82cc4e89bfc9ada83a1fd302 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5600989c2f634c4cb630839afa98c1a3" member_type: VOTER last_known_addr { host: "127.11.102.3" port: 43283 } } peers { permanent_uuid: "cfae6937c171468db2b68d87b1149dea" member_type: VOTER last_known_addr { host: "127.11.102.2" port: 40107 } } peers { permanent_uuid: "3fd44bea82cc4e89bfc9ada83a1fd302" member_type: VOTER last_known_addr { host: "127.11.102.1" port: 37171 } }
I20260620 06:10:34.371768 28634 leader_election.cc:288] T 47cd9415278242fbac0701a0a2e559f3 P 3fd44bea82cc4e89bfc9ada83a1fd302 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5600989c2f634c4cb630839afa98c1a3 (127.11.102.3:43283), cfae6937c171468db2b68d87b1149dea (127.11.102.2:40107)
W20260620 06:10:34.372577 28220 leader_election.cc:341] T 47cd9415278242fbac0701a0a2e559f3 P 3fd44bea82cc4e89bfc9ada83a1fd302 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 5600989c2f634c4cb630839afa98c1a3 (127.11.102.3:43283): Not found: Tablet not found: 47cd9415278242fbac0701a0a2e559f3
W20260620 06:10:34.372750 28221 leader_election.cc:341] T 47cd9415278242fbac0701a0a2e559f3 P 3fd44bea82cc4e89bfc9ada83a1fd302 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer cfae6937c171468db2b68d87b1149dea (127.11.102.2:40107): Not found: Tablet not found: 47cd9415278242fbac0701a0a2e559f3
I20260620 06:10:34.372812 28221 leader_election.cc:302] T 47cd9415278242fbac0701a0a2e559f3 P 3fd44bea82cc4e89bfc9ada83a1fd302 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3fd44bea82cc4e89bfc9ada83a1fd302; no voters: 5600989c2f634c4cb630839afa98c1a3, cfae6937c171468db2b68d87b1149dea
I20260620 06:10:34.373039 28634 raft_consensus.cc:2726] T 47cd9415278242fbac0701a0a2e559f3 P 3fd44bea82cc4e89bfc9ada83a1fd302 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 06:10:34.389377 28635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:10:34.388923 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.102.62:41831 (local address 127.11.102.2:38997)
0620 06:10:34.389221 (+ 298us) negotiation.cc:107] Waiting for socket to connect
0620 06:10:34.389285 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.102.62:41831: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":218,"thread_start_us":81,"threads_started":1}
I20260620 06:10:34.570770 28636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:10:34.570377 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.102.62:41831 (local address 127.11.102.3:54611)
0620 06:10:34.570633 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0620 06:10:34.570684 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.102.62:41831: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":89,"threads_started":1}
I20260620 06:10:35.213454 28637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:10:35.213045 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.102.62:41831 (local address 127.11.102.1:55623)
0620 06:10:35.213308 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0620 06:10:35.213362 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.102.62:41831: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":73,"threads_started":1}
I20260620 06:10:35.390587 28638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:10:35.390188 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.102.62:41831 (local address 127.11.102.2:41457)
0620 06:10:35.390444 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0620 06:10:35.390499 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.102.62:41831: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":84,"threads_started":1}
I20260620 06:10:35.571918 28639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:10:35.571428 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.102.62:41831 (local address 127.11.102.3:48943)
0620 06:10:35.571713 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0620 06:10:35.571787 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.102.62:41831: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":73,"threads_started":1}
I20260620 06:10:36.214668 28640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:10:36.214225 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.102.62:41831 (local address 127.11.102.1:47759)
0620 06:10:36.214521 (+ 296us) negotiation.cc:107] Waiting for socket to connect
0620 06:10:36.214574 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.102.62:41831: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":228,"thread_start_us":104,"threads_started":1}
I20260620 06:10:36.391747 28641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:10:36.391318 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.102.62:41831 (local address 127.11.102.2:36625)
0620 06:10:36.391571 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0620 06:10:36.391646 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.102.62:41831: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":87,"threads_started":1}
I20260620 06:10:36.572985 28642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:10:36.572603 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.102.62:41831 (local address 127.11.102.3:48701)
0620 06:10:36.572851 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0620 06:10:36.572901 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.102.62:41831: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":83,"threads_started":1}
I20260620 06:10:37.215898 28643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:10:37.215430 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.102.62:41831 (local address 127.11.102.1:52305)
0620 06:10:37.215745 (+ 315us) negotiation.cc:107] Waiting for socket to connect
0620 06:10:37.215803 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.102.62:41831: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":239,"thread_start_us":88,"threads_started":1}
I20260620 06:10:37.393002 28644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:10:37.392554 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.102.62:41831 (local address 127.11.102.2:36781)
0620 06:10:37.392838 (+ 284us) negotiation.cc:107] Waiting for socket to connect
0620 06:10:37.392902 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.102.62:41831: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":209,"thread_start_us":98,"threads_started":1}
I20260620 06:10:37.574239 28645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:10:37.573763 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.102.62:41831 (local address 127.11.102.3:49275)
0620 06:10:37.574056 (+ 293us) negotiation.cc:107] Waiting for socket to connect
0620 06:10:37.574139 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.102.62:41831: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":218,"thread_start_us":91,"threads_started":1}
I20260620 06:10:38.217044 28646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:10:38.216595 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.102.62:41831 (local address 127.11.102.1:54199)
0620 06:10:38.216885 (+ 290us) negotiation.cc:107] Waiting for socket to connect
0620 06:10:38.216946 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.102.62:41831: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":212,"thread_start_us":83,"threads_started":1}
I20260620 06:10:38.394435 28647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:10:38.393942 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.102.62:41831 (local address 127.11.102.2:51311)
0620 06:10:38.394265 (+ 323us) negotiation.cc:107] Waiting for socket to connect
0620 06:10:38.394342 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.102.62:41831: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":233,"thread_start_us":108,"threads_started":1}
I20260620 06:10:38.575381 28648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:10:38.574959 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.102.62:41831 (local address 127.11.102.3:45559)
0620 06:10:38.575229 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0620 06:10:38.575289 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.102.62:41831: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":76,"threads_started":1}
I20260620 06:10:39.218245 28649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:10:39.217817 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.102.62:41831 (local address 127.11.102.1:45855)
0620 06:10:39.218079 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0620 06:10:39.218148 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.102.62:41831: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":107,"threads_started":1}
I20260620 06:10:39.395890 28650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:10:39.395414 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.102.62:41831 (local address 127.11.102.2:51233)
0620 06:10:39.395728 (+ 314us) negotiation.cc:107] Waiting for socket to connect
0620 06:10:39.395792 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.102.62:41831: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":235,"thread_start_us":90,"threads_started":1}
I20260620 06:10:39.576678 28651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:10:39.576240 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.102.62:41831 (local address 127.11.102.3:37665)
0620 06:10:39.576508 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0620 06:10:39.576573 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.102.62:41831: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":78,"threads_started":1}
I20260620 06:10:40.087256 28652 raft_consensus.cc:482] T 47cd9415278242fbac0701a0a2e559f3 P 3fd44bea82cc4e89bfc9ada83a1fd302 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 06:10:40.087401 28652 raft_consensus.cc:504] T 47cd9415278242fbac0701a0a2e559f3 P 3fd44bea82cc4e89bfc9ada83a1fd302 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5600989c2f634c4cb630839afa98c1a3" member_type: VOTER last_known_addr { host: "127.11.102.3" port: 43283 } } peers { permanent_uuid: "cfae6937c171468db2b68d87b1149dea" member_type: VOTER last_known_addr { host: "127.11.102.2" port: 40107 } } peers { permanent_uuid: "3fd44bea82cc4e89bfc9ada83a1fd302" member_type: VOTER last_known_addr { host: "127.11.102.1" port: 37171 } }
I20260620 06:10:40.087925 28652 leader_election.cc:288] T 47cd9415278242fbac0701a0a2e559f3 P 3fd44bea82cc4e89bfc9ada83a1fd302 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5600989c2f634c4cb630839afa98c1a3 (127.11.102.3:43283), cfae6937c171468db2b68d87b1149dea (127.11.102.2:40107)
W20260620 06:10:40.088755 28221 leader_election.cc:341] T 47cd9415278242fbac0701a0a2e559f3 P 3fd44bea82cc4e89bfc9ada83a1fd302 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer cfae6937c171468db2b68d87b1149dea (127.11.102.2:40107): Not found: Tablet not found: 47cd9415278242fbac0701a0a2e559f3
W20260620 06:10:40.088857 28220 leader_election.cc:341] T 47cd9415278242fbac0701a0a2e559f3 P 3fd44bea82cc4e89bfc9ada83a1fd302 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 5600989c2f634c4cb630839afa98c1a3 (127.11.102.3:43283): Not found: Tablet not found: 47cd9415278242fbac0701a0a2e559f3
I20260620 06:10:40.088968 28220 leader_election.cc:302] T 47cd9415278242fbac0701a0a2e559f3 P 3fd44bea82cc4e89bfc9ada83a1fd302 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3fd44bea82cc4e89bfc9ada83a1fd302; no voters: 5600989c2f634c4cb630839afa98c1a3, cfae6937c171468db2b68d87b1149dea
I20260620 06:10:40.089124 28652 raft_consensus.cc:2726] T 47cd9415278242fbac0701a0a2e559f3 P 3fd44bea82cc4e89bfc9ada83a1fd302 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 06:10:40.219878 28653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:10:40.219314 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.102.62:41831 (local address 127.11.102.1:35583)
0620 06:10:40.219671 (+ 357us) negotiation.cc:107] Waiting for socket to connect
0620 06:10:40.219767 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.102.62:41831: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":207,"thread_start_us":80,"threads_started":1}
I20260620 06:10:40.397114 28654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:10:40.396734 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.102.62:41831 (local address 127.11.102.2:55571)
0620 06:10:40.396973 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0620 06:10:40.397025 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.102.62:41831: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":173,"thread_start_us":85,"threads_started":1}
I20260620 06:10:40.568780 11672 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9HMp43/build/debug/bin/kudu with pid 28163
I20260620 06:10:40.574601 11672 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9HMp43/build/debug/bin/kudu with pid 28336
I20260620 06:10:40.578023 28655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:10:40.577605 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.102.62:41831 (local address 127.11.102.3:35485)
0620 06:10:40.577865 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0620 06:10:40.577934 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.102.62:41831: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":82,"threads_started":1}
I20260620 06:10:40.580245 11672 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9HMp43/build/debug/bin/kudu with pid 28470
2026-06-20T06:10:40Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (59344 ms)
[----------] 30 tests from SecurityITest (237454 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781935603120653-11672-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 06:10:40 dist-test-slave-9hm5 krb5kdc[28658](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 06:10:40 dist-test-slave-9hm5 krb5kdc[28658](info): set up 2 sockets
Jun 20 06:10:40 dist-test-slave-9hm5 krb5kdc[28658](info): commencing operation
krb5kdc: starting...
W20260620 06:10:41.631374 11672 mini_kdc.cc:121] Time spent starting KDC: real 1.029s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781935603120653-11672-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781935603120653-11672-0/krb5kdc/test-user.keytab.
Jun 20 06:10:41 dist-test-slave-9hm5 krb5kdc[28658](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935841, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T06:10:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T06:10:41Z Disabled control of system clock
WARNING: no policy specified for kudu/127.11.102.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.102.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.102.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781935603120653-11672-0/krb5kdc/kudu_127.11.102.62.keytab.
Entry for principal kudu/127.11.102.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781935603120653-11672-0/krb5kdc/kudu_127.11.102.62.keytab.
I20260620 06:10:41.754365 11672 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9HMp43/build/debug/bin/kudu
/tmp/dist-test-task9HMp43/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781935603120653-11672-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781935603120653-11672-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781935603120653-11672-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781935603120653-11672-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.102.62:34917
--webserver_interface=127.11.102.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781935603120653-11672-0/krb5kdc/kudu_127.11.102.62.keytab
--principal=kudu/127.11.102.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.102.20:35385
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.102.62:34917
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781935603120653-11672-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781935603120653-11672-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781935603120653-11672-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 06:10:41.866761 28671 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:10:41.867101 28671 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:10:41.867187 28671 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:10:41.871289 28671 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 06:10:41.871393 28671 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:10:41.871439 28671 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:10:41.871481 28671 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 06:10:41.871516 28671 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 06:10:41.876999 28671 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.102.20:35385
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781935603120653-11672-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781935603120653-11672-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.102.62:34917
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781935603120653-11672-0/krb5kdc/kudu_127.11.102.62.keytab
--principal=kudu/127.11.102.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.102.62:34917
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781935603120653-11672-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.102.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28671
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781935603120653-11672-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 05:28:48 UTC on ccycloud.quasar-fixhms.root.comops.site
build id 4457020
I20260620 06:10:41.878176 28671 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:10:41.879335 28671 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 06:10:41.888264 28671 server_base.cc:1034] running on GCE node
W20260620 06:10:41.888268 28679 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:10:41.888327 28677 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:10:41.888327 28676 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:10:41.889299 28671 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:10:41.890537 28671 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:10:41.891709 28671 hybrid_clock.cc:648] HybridClock initialized: now 1781935841891674 us; error 48 us; skew 500 ppm
Jun 20 06:10:41 dist-test-slave-9hm5 krb5kdc[28658](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935841, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.102.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 06:10:41.894493 28671 init.cc:377] Logged in from keytab as kudu/127.11.102.62@KRBTEST.COM (short username kudu)
I20260620 06:10:41.895378 28671 webserver.cc:460] Webserver started at http://127.11.102.62:34991/ using document root <none> and password file <none>
I20260620 06:10:41.895993 28671 fs_manager.cc:359] Metadata directory not provided
I20260620 06:10:41.896080 28671 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:10:41.896310 28671 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 06:10:41.898375 28671 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781935603120653-11672-0/minicluster-data/master-0/data/instance:
uuid: "7fc9e7ca1b284980816a651a762ff0ff"
format_stamp: "Formatted at 2026-06-20 06:10:41 on dist-test-slave-9hm5"
server_key: "bd02ea52ff17180facd9d03c263df6a2"
server_key_iv: "2331221f44a89907569bf712ed9e62f5"
server_key_version: "encryptionkey@0"
I20260620 06:10:41.898957 28671 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781935603120653-11672-0/minicluster-data/master-0/wal/instance:
uuid: "7fc9e7ca1b284980816a651a762ff0ff"
format_stamp: "Formatted at 2026-06-20 06:10:41 on dist-test-slave-9hm5"
server_key: "bd02ea52ff17180facd9d03c263df6a2"
server_key_iv: "2331221f44a89907569bf712ed9e62f5"
server_key_version: "encryptionkey@0"
I20260620 06:10:41.902717 28671 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260620 06:10:41.905120 28686 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:10:41.906327 28671 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260620 06:10:41.906456 28671 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781935603120653-11672-0/minicluster-data/master-0/data,/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781935603120653-11672-0/minicluster-data/master-0/wal
uuid: "7fc9e7ca1b284980816a651a762ff0ff"
format_stamp: "Formatted at 2026-06-20 06:10:41 on dist-test-slave-9hm5"
server_key: "bd02ea52ff17180facd9d03c263df6a2"
server_key_iv: "2331221f44a89907569bf712ed9e62f5"
server_key_version: "encryptionkey@0"
I20260620 06:10:41.906564 28671 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781935603120653-11672-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781935603120653-11672-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781935603120653-11672-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:10:41.929283 28671 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:10:41.932045 28671 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:10:41.932240 28671 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:10:41.939070 28671 rpc_server.cc:307] RPC server started. Bound to: 127.11.102.62:34917
I20260620 06:10:41.939097 28738 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.102.62:34917 every 8 connection(s)
I20260620 06:10:41.939978 28671 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781935603120653-11672-0/minicluster-data/master-0/data/info.pb
I20260620 06:10:41.940482 11672 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9HMp43/build/debug/bin/kudu as pid 28671
I20260620 06:10:41.940630 11672 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781935603120653-11672-0/minicluster-data/master-0/wal/instance
I20260620 06:10:41.940937 11672 external_mini_cluster.cc:1428] Setting key 9728c078d53d322586f3fa160c17dc88
I20260620 06:10:41.942960 28739 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 06:10:41.949182 28739 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7fc9e7ca1b284980816a651a762ff0ff: Bootstrap starting.
Jun 20 06:10:41 dist-test-slave-9hm5 krb5kdc[28658](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935841, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.102.62@KRBTEST.COM
I20260620 06:10:41.951263 28739 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7fc9e7ca1b284980816a651a762ff0ff: Neither blocks nor log segments found. Creating new log.
I20260620 06:10:41.951999 28739 log.cc:824] T 00000000000000000000000000000000 P 7fc9e7ca1b284980816a651a762ff0ff: Log is configured to *not* fsync() on all Append() calls
I20260620 06:10:41.953677 28739 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7fc9e7ca1b284980816a651a762ff0ff: No bootstrap required, opened a new log
I20260620 06:10:41.954660 28742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:10:41.942302 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40716 (local address 127.11.102.62:34917)
0620 06:10:41.942695 (+ 393us) server_negotiation.cc:206] Beginning negotiation
0620 06:10:41.942713 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0620 06:10:41.942755 (+ 42us) server_negotiation.cc:407] Connection header received
0620 06:10:41.943413 (+ 658us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:10:41.943429 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:10:41.943791 (+ 362us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:10:41.944070 (+ 279us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:10:41.944962 (+ 892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:10:41.945922 (+ 960us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:10:41.946778 (+ 856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:10:41.947282 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:10:41.947448 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:10:41.950258 (+ 2810us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:10:41.950263 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:10:41.950274 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:10:41.950320 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:10:41.952049 (+ 1729us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:10:41.952470 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:10:41.952480 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:10:41.952485 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:10:41.952590 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:10:41.952870 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:10:41.952874 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:10:41.952876 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:10:41.953232 (+ 356us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:10:41.953424 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0620 06:10:41.953684 (+ 260us) server_negotiation.cc:299] Negotiation successful
0620 06:10:41.954001 (+ 317us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":259,"thread_start_us":144,"threads_started":1}
I20260620 06:10:41.957346 28739 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7fc9e7ca1b284980816a651a762ff0ff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7fc9e7ca1b284980816a651a762ff0ff" member_type: VOTER last_known_addr { host: "127.11.102.62" port: 34917 } }
I20260620 06:10:41.957566 28739 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7fc9e7ca1b284980816a651a762ff0ff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 06:10:41.957638 28739 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7fc9e7ca1b284980816a651a762ff0ff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7fc9e7ca1b284980816a651a762ff0ff, State: Initialized, Role: FOLLOWER
I20260620 06:10:41.958035 28739 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7fc9e7ca1b284980816a651a762ff0ff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7fc9e7ca1b284980816a651a762ff0ff" member_type: VOTER last_known_addr { host: "127.11.102.62" port: 34917 } }
I20260620 06:10:41.958176 28739 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7fc9e7ca1b284980816a651a762ff0ff [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 06:10:41.958256 28739 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7fc9e7ca1b284980816a651a762ff0ff [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 06:10:41.958366 28739 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7fc9e7ca1b284980816a651a762ff0ff [term 0 FOLLOWER]: Advancing to term 1
I20260620 06:10:41.959203 28739 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7fc9e7ca1b284980816a651a762ff0ff [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7fc9e7ca1b284980816a651a762ff0ff" member_type: VOTER last_known_addr { host: "127.11.102.62" port: 34917 } }
I20260620 06:10:41.959558 28739 leader_election.cc:302] T 00000000000000000000000000000000 P 7fc9e7ca1b284980816a651a762ff0ff [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7fc9e7ca1b284980816a651a762ff0ff; no voters:
I20260620 06:10:41.959896 28739 leader_election.cc:288] T 00000000000000000000000000000000 P 7fc9e7ca1b284980816a651a762ff0ff [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 06:10:41.960006 28744 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7fc9e7ca1b284980816a651a762ff0ff [term 1 FOLLOWER]: Leader election won for term 1
I20260620 06:10:41.960239 28744 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7fc9e7ca1b284980816a651a762ff0ff [term 1 LEADER]: Becoming Leader. State: Replica: 7fc9e7ca1b284980816a651a762ff0ff, State: Running, Role: LEADER
I20260620 06:10:41.960611 28744 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7fc9e7ca1b284980816a651a762ff0ff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7fc9e7ca1b284980816a651a762ff0ff" member_type: VOTER last_known_addr { host: "127.11.102.62" port: 34917 } }
I20260620 06:10:41.961108 28739 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7fc9e7ca1b284980816a651a762ff0ff [sys.catalog]: configured and running, proceeding with master startup.
I20260620 06:10:41.962234 28745 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7fc9e7ca1b284980816a651a762ff0ff [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7fc9e7ca1b284980816a651a762ff0ff" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7fc9e7ca1b284980816a651a762ff0ff" member_type: VOTER last_known_addr { host: "127.11.102.62" port: 34917 } } }
I20260620 06:10:41.962386 28745 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7fc9e7ca1b284980816a651a762ff0ff [sys.catalog]: This master's current role is: LEADER
I20260620 06:10:41.962778 28753 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 06:10:41.962759 28746 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7fc9e7ca1b284980816a651a762ff0ff [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7fc9e7ca1b284980816a651a762ff0ff. Latest consensus state: current_term: 1 leader_uuid: "7fc9e7ca1b284980816a651a762ff0ff" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7fc9e7ca1b284980816a651a762ff0ff" member_type: VOTER last_known_addr { host: "127.11.102.62" port: 34917 } } }
I20260620 06:10:41.963238 28746 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7fc9e7ca1b284980816a651a762ff0ff [sys.catalog]: This master's current role is: LEADER
I20260620 06:10:41.965699 28753 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 06:10:41.970757 28753 catalog_manager.cc:1347] Generated new cluster ID: efbe0171d0f448498c36d22627001305
I20260620 06:10:41.970820 28753 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 06:10:41.985651 28753 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 06:10:41.986652 28753 catalog_manager.cc:1504] Loading token signing keys...
I20260620 06:10:41.996524 28753 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7fc9e7ca1b284980816a651a762ff0ff: Generated new TSK 0
I20260620 06:10:41.997228 28753 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.11.102.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.102.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.102.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781935603120653-11672-0/krb5kdc/kudu_127.11.102.1.keytab.
Entry for principal kudu/127.11.102.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781935603120653-11672-0/krb5kdc/kudu_127.11.102.1.keytab.
I20260620 06:10:42.030084 11672 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9HMp43/build/debug/bin/kudu
/tmp/dist-test-task9HMp43/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781935603120653-11672-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781935603120653-11672-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781935603120653-11672-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781935603120653-11672-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.102.1:0
--local_ip_for_outbound_sockets=127.11.102.1
--webserver_interface=127.11.102.1
--webserver_port=0
--tserver_master_addrs=127.11.102.62:34917
--keytab_file=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781935603120653-11672-0/krb5kdc/kudu_127.11.102.1.keytab
--principal=kudu/127.11.102.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.102.20:35385
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781935603120653-11672-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781935603120653-11672-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781935603120653-11672-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 06:10:42.138702 28765 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:10:42.138960 28765 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:10:42.139051 28765 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:10:42.143038 28765 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:10:42.143090 28765 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:10:42.143193 28765 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.102.1
I20260620 06:10:42.148530 28765 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.102.20:35385
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781935603120653-11672-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781935603120653-11672-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781935603120653-11672-0/krb5kdc/kudu_127.11.102.1.keytab
--principal=kudu/127.11.102.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.102.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781935603120653-11672-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.11.102.1
--webserver_port=0
--tserver_master_addrs=127.11.102.62:34917
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28765
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.102.1
--log_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781935603120653-11672-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 05:28:48 UTC on ccycloud.quasar-fixhms.root.comops.site
build id 4457020
I20260620 06:10:42.149499 28765 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:10:42.150466 28765 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 06:10:42.159721 28770 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:10:42.159791 28765 server_base.cc:1034] running on GCE node
W20260620 06:10:42.159734 28773 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:10:42.159721 28771 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:10:42.160310 28765 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:10:42.160918 28765 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:10:42.162072 28765 hybrid_clock.cc:648] HybridClock initialized: now 1781935842162059 us; error 44 us; skew 500 ppm
Jun 20 06:10:42 dist-test-slave-9hm5 krb5kdc[28658](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935842, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.102.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 06:10:42.165023 28765 init.cc:377] Logged in from keytab as kudu/127.11.102.1@KRBTEST.COM (short username kudu)
I20260620 06:10:42.165880 28765 webserver.cc:460] Webserver started at http://127.11.102.1:41721/ using document root <none> and password file <none>
I20260620 06:10:42.166425 28765 fs_manager.cc:359] Metadata directory not provided
I20260620 06:10:42.166484 28765 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:10:42.166702 28765 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 06:10:42.168758 28765 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781935603120653-11672-0/minicluster-data/ts-0/data/instance:
uuid: "f75880b31b994309ad340701e0fc9237"
format_stamp: "Formatted at 2026-06-20 06:10:42 on dist-test-slave-9hm5"
server_key: "a3df5633473e6c7eb449ba49d63b95e6"
server_key_iv: "f2930971324a698e83c8538df49e518a"
server_key_version: "encryptionkey@0"
I20260620 06:10:42.169332 28765 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781935603120653-11672-0/minicluster-data/ts-0/wal/instance:
uuid: "f75880b31b994309ad340701e0fc9237"
format_stamp: "Formatted at 2026-06-20 06:10:42 on dist-test-slave-9hm5"
server_key: "a3df5633473e6c7eb449ba49d63b95e6"
server_key_iv: "f2930971324a698e83c8538df49e518a"
server_key_version: "encryptionkey@0"
I20260620 06:10:42.173027 28765 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 06:10:42.175354 28780 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:10:42.176559 28765 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 06:10:42.176680 28765 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781935603120653-11672-0/minicluster-data/ts-0/data,/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781935603120653-11672-0/minicluster-data/ts-0/wal
uuid: "f75880b31b994309ad340701e0fc9237"
format_stamp: "Formatted at 2026-06-20 06:10:42 on dist-test-slave-9hm5"
server_key: "a3df5633473e6c7eb449ba49d63b95e6"
server_key_iv: "f2930971324a698e83c8538df49e518a"
server_key_version: "encryptionkey@0"
I20260620 06:10:42.176784 28765 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781935603120653-11672-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781935603120653-11672-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781935603120653-11672-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:10:42.197232 28765 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:10:42.200076 28765 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:10:42.200275 28765 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:10:42.200843 28765 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 06:10:42.201648 28765 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 06:10:42.201714 28765 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 06:10:42.201782 28765 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 06:10:42.201829 28765 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 06:10:42.211098 28765 rpc_server.cc:307] RPC server started. Bound to: 127.11.102.1:35423
I20260620 06:10:42.211143 28893 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.102.1:35423 every 8 connection(s)
I20260620 06:10:42.212045 28765 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781935603120653-11672-0/minicluster-data/ts-0/data/info.pb
I20260620 06:10:42.216322 11672 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9HMp43/build/debug/bin/kudu as pid 28765
I20260620 06:10:42.216462 11672 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781935603120653-11672-0/minicluster-data/ts-0/wal/instance
I20260620 06:10:42.216799 11672 external_mini_cluster.cc:1428] Setting key 89f57c196d1446549e639063fc11bfcc
Jun 20 06:10:42 dist-test-slave-9hm5 krb5kdc[28658](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935842, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.102.1@KRBTEST.COM for kudu/127.11.102.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.11.102.2@KRBTEST.COM; defaulting to no policy
I20260620 06:10:42.227496 28742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:10:42.213867 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.102.1:42593 (local address 127.11.102.62:34917)
0620 06:10:42.214004 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0620 06:10:42.214008 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 06:10:42.214997 (+ 989us) server_negotiation.cc:407] Connection header received
0620 06:10:42.215960 (+ 963us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:10:42.215963 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:10:42.216008 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:10:42.216081 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:10:42.217665 (+ 1584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:10:42.218407 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:10:42.220209 (+ 1802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:10:42.220624 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:10:42.220751 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:10:42.223429 (+ 2678us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:10:42.223433 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:10:42.223435 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:10:42.223459 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:10:42.224989 (+ 1530us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:10:42.225675 (+ 686us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:10:42.225678 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:10:42.225681 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:10:42.225800 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:10:42.226264 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:10:42.226269 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:10:42.226272 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:10:42.226592 (+ 320us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:10:42.226700 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0620 06:10:42.227239 (+ 539us) server_negotiation.cc:299] Negotiation successful
0620 06:10:42.227370 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260620 06:10:42.228363 28896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:10:42.214237 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.102.62:34917 (local address 127.11.102.1:42593)
0620 06:10:42.214870 (+ 633us) negotiation.cc:107] Waiting for socket to connect
0620 06:10:42.214910 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0620 06:10:42.215768 (+ 858us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:10:42.216319 (+ 551us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:10:42.216330 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:10:42.216851 (+ 521us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:10:42.217509 (+ 658us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:10:42.217532 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:10:42.218529 (+ 997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:10:42.218533 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:10:42.220065 (+ 1532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:10:42.220080 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:10:42.220767 (+ 687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:10:42.220770 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:10:42.220916 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:10:42.221515 (+ 599us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:10:42.221540 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:10:42.223255 (+ 1715us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:10:42.225104 (+ 1849us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:10:42.225110 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:10:42.225126 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:10:42.225548 (+ 422us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:10:42.225918 (+ 370us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:10:42.225923 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:10:42.225926 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:10:42.226137 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:10:42.226709 (+ 572us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:10:42.226714 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:10:42.226974 (+ 260us) client_negotiation.cc:769] Sending connection context
0620 06:10:42.227163 (+ 189us) client_negotiation.cc:240] Negotiation successful
0620 06:10:42.227428 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":445,"thread_start_us":114,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.102.2@KRBTEST.COM" created.
I20260620 06:10:42.229753 28894 heartbeater.cc:347] Connected to a master server at 127.11.102.62:34917
I20260620 06:10:42.230031 28894 heartbeater.cc:464] Registering TS with master...
I20260620 06:10:42.230541 28894 heartbeater.cc:511] Master 127.11.102.62:34917 requested a full tablet report, sending...
I20260620 06:10:42.232182 28703 ts_manager.cc:194] Registered new tserver with Master: f75880b31b994309ad340701e0fc9237 (127.11.102.1:35423)
I20260620 06:10:42.233331 28703 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.102.1@KRBTEST.COM'} at 127.11.102.1:42593
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.102.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781935603120653-11672-0/krb5kdc/kudu_127.11.102.2.keytab.
Entry for principal kudu/127.11.102.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781935603120653-11672-0/krb5kdc/kudu_127.11.102.2.keytab.
I20260620 06:10:42.246258 11672 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9HMp43/build/debug/bin/kudu
/tmp/dist-test-task9HMp43/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781935603120653-11672-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781935603120653-11672-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781935603120653-11672-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781935603120653-11672-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.102.2:0
--local_ip_for_outbound_sockets=127.11.102.2
--webserver_interface=127.11.102.2
--webserver_port=0
--tserver_master_addrs=127.11.102.62:34917
--keytab_file=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781935603120653-11672-0/krb5kdc/kudu_127.11.102.2.keytab
--principal=kudu/127.11.102.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.102.20:35385
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781935603120653-11672-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781935603120653-11672-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781935603120653-11672-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 06:10:42.354413 28899 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:10:42.354682 28899 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:10:42.354751 28899 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:10:42.358801 28899 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:10:42.358884 28899 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:10:42.358973 28899 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.102.2
I20260620 06:10:42.364393 28899 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.102.20:35385
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781935603120653-11672-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781935603120653-11672-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781935603120653-11672-0/krb5kdc/kudu_127.11.102.2.keytab
--principal=kudu/127.11.102.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.102.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781935603120653-11672-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.11.102.2
--webserver_port=0
--tserver_master_addrs=127.11.102.62:34917
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28899
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.102.2
--log_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781935603120653-11672-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 05:28:48 UTC on ccycloud.quasar-fixhms.root.comops.site
build id 4457020
I20260620 06:10:42.365428 28899 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:10:42.366465 28899 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 06:10:42.375878 28905 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:10:42.375926 28904 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:10:42.375916 28907 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:10:42.376034 28899 server_base.cc:1034] running on GCE node
I20260620 06:10:42.376430 28899 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:10:42.377074 28899 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:10:42.378250 28899 hybrid_clock.cc:648] HybridClock initialized: now 1781935842378234 us; error 28 us; skew 500 ppm
Jun 20 06:10:42 dist-test-slave-9hm5 krb5kdc[28658](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935842, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.102.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 06:10:42.381028 28899 init.cc:377] Logged in from keytab as kudu/127.11.102.2@KRBTEST.COM (short username kudu)
I20260620 06:10:42.381922 28899 webserver.cc:460] Webserver started at http://127.11.102.2:45355/ using document root <none> and password file <none>
I20260620 06:10:42.382460 28899 fs_manager.cc:359] Metadata directory not provided
I20260620 06:10:42.382548 28899 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:10:42.382766 28899 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 06:10:42.384768 28899 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781935603120653-11672-0/minicluster-data/ts-1/data/instance:
uuid: "7040f5853eba4bf6967fda1b4952f45e"
format_stamp: "Formatted at 2026-06-20 06:10:42 on dist-test-slave-9hm5"
server_key: "c63f418d594622afb21085bfc1255685"
server_key_iv: "b41c45f92111e2d79f259209b2b7fd60"
server_key_version: "encryptionkey@0"
I20260620 06:10:42.385331 28899 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781935603120653-11672-0/minicluster-data/ts-1/wal/instance:
uuid: "7040f5853eba4bf6967fda1b4952f45e"
format_stamp: "Formatted at 2026-06-20 06:10:42 on dist-test-slave-9hm5"
server_key: "c63f418d594622afb21085bfc1255685"
server_key_iv: "b41c45f92111e2d79f259209b2b7fd60"
server_key_version: "encryptionkey@0"
I20260620 06:10:42.388882 28899 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 06:10:42.391312 28914 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:10:42.392621 28899 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 06:10:42.392748 28899 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781935603120653-11672-0/minicluster-data/ts-1/data,/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781935603120653-11672-0/minicluster-data/ts-1/wal
uuid: "7040f5853eba4bf6967fda1b4952f45e"
format_stamp: "Formatted at 2026-06-20 06:10:42 on dist-test-slave-9hm5"
server_key: "c63f418d594622afb21085bfc1255685"
server_key_iv: "b41c45f92111e2d79f259209b2b7fd60"
server_key_version: "encryptionkey@0"
I20260620 06:10:42.392854 28899 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781935603120653-11672-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781935603120653-11672-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781935603120653-11672-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:10:42.418565 28899 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:10:42.421945 28899 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:10:42.422250 28899 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:10:42.422923 28899 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 06:10:42.423796 28899 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 06:10:42.423846 28899 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 06:10:42.423897 28899 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 06:10:42.423913 28899 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 06:10:42.432731 28899 rpc_server.cc:307] RPC server started. Bound to: 127.11.102.2:34811
I20260620 06:10:42.432777 29027 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.102.2:34811 every 8 connection(s)
I20260620 06:10:42.433665 28899 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781935603120653-11672-0/minicluster-data/ts-1/data/info.pb
Jun 20 06:10:42 dist-test-slave-9hm5 krb5kdc[28658](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935842, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.102.2@KRBTEST.COM for kudu/127.11.102.62@KRBTEST.COM
I20260620 06:10:42.442830 11672 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9HMp43/build/debug/bin/kudu as pid 28899
I20260620 06:10:42.442970 11672 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781935603120653-11672-0/minicluster-data/ts-1/wal/instance
I20260620 06:10:42.443336 11672 external_mini_cluster.cc:1428] Setting key ec156ba7736c0885983aaf95eb0f7caf
I20260620 06:10:42.446866 28742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:10:42.435457 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.102.2:59021 (local address 127.11.102.62:34917)
0620 06:10:42.435597 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0620 06:10:42.435600 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 06:10:42.436675 (+ 1075us) server_negotiation.cc:407] Connection header received
0620 06:10:42.437516 (+ 841us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:10:42.437519 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:10:42.437566 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:10:42.437643 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:10:42.438744 (+ 1101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:10:42.439262 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:10:42.440398 (+ 1136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:10:42.440607 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:10:42.440680 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:10:42.442908 (+ 2228us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:10:42.442912 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:10:42.442917 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:10:42.442949 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:10:42.444712 (+ 1763us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:10:42.445238 (+ 526us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:10:42.445269 (+ 31us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:10:42.445271 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:10:42.445375 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:10:42.445698 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:10:42.445705 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:10:42.445710 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:10:42.445981 (+ 271us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:10:42.446098 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0620 06:10:42.446537 (+ 439us) server_negotiation.cc:299] Negotiation successful
0620 06:10:42.446689 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260620 06:10:42.447956 29030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:10:42.435905 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.102.62:34917 (local address 127.11.102.2:59021)
0620 06:10:42.436531 (+ 626us) negotiation.cc:107] Waiting for socket to connect
0620 06:10:42.436572 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0620 06:10:42.437330 (+ 758us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:10:42.437805 (+ 475us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:10:42.437812 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:10:42.438162 (+ 350us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:10:42.438624 (+ 462us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:10:42.438636 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:10:42.439361 (+ 725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:10:42.439364 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:10:42.440289 (+ 925us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:10:42.440298 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:10:42.440726 (+ 428us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:10:42.440728 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:10:42.440857 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:10:42.441375 (+ 518us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:10:42.441390 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:10:42.442744 (+ 1354us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:10:42.444833 (+ 2089us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:10:42.444838 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:10:42.444854 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:10:42.445134 (+ 280us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:10:42.445476 (+ 342us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:10:42.445478 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:10:42.445479 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:10:42.445605 (+ 126us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:10:42.446083 (+ 478us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:10:42.446088 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:10:42.446298 (+ 210us) client_negotiation.cc:769] Sending connection context
0620 06:10:42.446456 (+ 158us) client_negotiation.cc:240] Negotiation successful
0620 06:10:42.446829 (+ 373us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":441,"thread_start_us":113,"threads_started":1}
I20260620 06:10:42.449126 29028 heartbeater.cc:347] Connected to a master server at 127.11.102.62:34917
I20260620 06:10:42.449381 29028 heartbeater.cc:464] Registering TS with master...
I20260620 06:10:42.449878 29028 heartbeater.cc:511] Master 127.11.102.62:34917 requested a full tablet report, sending...
I20260620 06:10:42.450965 28703 ts_manager.cc:194] Registered new tserver with Master: 7040f5853eba4bf6967fda1b4952f45e (127.11.102.2:34811)
I20260620 06:10:42.451539 28703 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.102.2@KRBTEST.COM'} at 127.11.102.2:59021
WARNING: no policy specified for kudu/127.11.102.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.102.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.102.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781935603120653-11672-0/krb5kdc/kudu_127.11.102.3.keytab.
Entry for principal kudu/127.11.102.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781935603120653-11672-0/krb5kdc/kudu_127.11.102.3.keytab.
I20260620 06:10:42.470606 11672 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9HMp43/build/debug/bin/kudu
/tmp/dist-test-task9HMp43/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781935603120653-11672-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781935603120653-11672-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781935603120653-11672-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781935603120653-11672-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.102.3:0
--local_ip_for_outbound_sockets=127.11.102.3
--webserver_interface=127.11.102.3
--webserver_port=0
--tserver_master_addrs=127.11.102.62:34917
--keytab_file=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781935603120653-11672-0/krb5kdc/kudu_127.11.102.3.keytab
--principal=kudu/127.11.102.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.102.20:35385
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781935603120653-11672-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781935603120653-11672-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781935603120653-11672-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 06:10:42.584112 29033 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:10:42.584379 29033 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:10:42.584447 29033 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:10:42.588495 29033 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:10:42.588563 29033 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:10:42.588646 29033 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.102.3
I20260620 06:10:42.593688 29033 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.102.20:35385
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781935603120653-11672-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781935603120653-11672-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781935603120653-11672-0/krb5kdc/kudu_127.11.102.3.keytab
--principal=kudu/127.11.102.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.102.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781935603120653-11672-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.11.102.3
--webserver_port=0
--tserver_master_addrs=127.11.102.62:34917
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29033
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.102.3
--log_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781935603120653-11672-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 05:28:48 UTC on ccycloud.quasar-fixhms.root.comops.site
build id 4457020
I20260620 06:10:42.594686 29033 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:10:42.595737 29033 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 06:10:42.605175 29033 server_base.cc:1034] running on GCE node
W20260620 06:10:42.605126 29039 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:10:42.605152 29038 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:10:42.605126 29041 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:10:42.605746 29033 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:10:42.606344 29033 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:10:42.607535 29033 hybrid_clock.cc:648] HybridClock initialized: now 1781935842607503 us; error 45 us; skew 500 ppm
Jun 20 06:10:42 dist-test-slave-9hm5 krb5kdc[28658](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935842, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.102.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 06:10:42.610113 29033 init.cc:377] Logged in from keytab as kudu/127.11.102.3@KRBTEST.COM (short username kudu)
I20260620 06:10:42.611037 29033 webserver.cc:460] Webserver started at http://127.11.102.3:41965/ using document root <none> and password file <none>
I20260620 06:10:42.611682 29033 fs_manager.cc:359] Metadata directory not provided
I20260620 06:10:42.611747 29033 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:10:42.611963 29033 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 06:10:42.613903 29033 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781935603120653-11672-0/minicluster-data/ts-2/data/instance:
uuid: "2ff12866e4c34bb1a57fdc24772fca03"
format_stamp: "Formatted at 2026-06-20 06:10:42 on dist-test-slave-9hm5"
server_key: "74ad064c60e366992ec5d0ffb379f658"
server_key_iv: "d088349579dd72ffa13ad9ec399ea44b"
server_key_version: "encryptionkey@0"
I20260620 06:10:42.614449 29033 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781935603120653-11672-0/minicluster-data/ts-2/wal/instance:
uuid: "2ff12866e4c34bb1a57fdc24772fca03"
format_stamp: "Formatted at 2026-06-20 06:10:42 on dist-test-slave-9hm5"
server_key: "74ad064c60e366992ec5d0ffb379f658"
server_key_iv: "d088349579dd72ffa13ad9ec399ea44b"
server_key_version: "encryptionkey@0"
I20260620 06:10:42.618041 29033 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260620 06:10:42.620371 29048 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:10:42.621481 29033 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 06:10:42.621604 29033 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781935603120653-11672-0/minicluster-data/ts-2/data,/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781935603120653-11672-0/minicluster-data/ts-2/wal
uuid: "2ff12866e4c34bb1a57fdc24772fca03"
format_stamp: "Formatted at 2026-06-20 06:10:42 on dist-test-slave-9hm5"
server_key: "74ad064c60e366992ec5d0ffb379f658"
server_key_iv: "d088349579dd72ffa13ad9ec399ea44b"
server_key_version: "encryptionkey@0"
I20260620 06:10:42.621702 29033 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781935603120653-11672-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781935603120653-11672-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781935603120653-11672-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:10:42.639253 29033 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:10:42.642305 29033 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:10:42.642493 29033 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:10:42.643072 29033 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 06:10:42.643898 29033 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 06:10:42.643963 29033 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 06:10:42.644025 29033 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 06:10:42.644070 29033 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 06:10:42.652769 29033 rpc_server.cc:307] RPC server started. Bound to: 127.11.102.3:41009
I20260620 06:10:42.652802 29161 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.102.3:41009 every 8 connection(s)
I20260620 06:10:42.653611 29033 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781935603120653-11672-0/minicluster-data/ts-2/data/info.pb
I20260620 06:10:42.656697 11672 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9HMp43/build/debug/bin/kudu as pid 29033
I20260620 06:10:42.656841 11672 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781935603120653-11672-0/minicluster-data/ts-2/wal/instance
I20260620 06:10:42.657135 11672 external_mini_cluster.cc:1428] Setting key 5e872c664ac94cb304effad59953dc72
Jun 20 06:10:42 dist-test-slave-9hm5 krb5kdc[28658](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935842, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.102.3@KRBTEST.COM for kudu/127.11.102.62@KRBTEST.COM
I20260620 06:10:42.666743 28742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:10:42.655341 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.102.3:47419 (local address 127.11.102.62:34917)
0620 06:10:42.655480 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0620 06:10:42.655484 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 06:10:42.656520 (+ 1036us) server_negotiation.cc:407] Connection header received
0620 06:10:42.657491 (+ 971us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:10:42.657494 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:10:42.657541 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:10:42.657614 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:10:42.659303 (+ 1689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:10:42.659883 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:10:42.661028 (+ 1145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:10:42.661241 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:10:42.661318 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:10:42.663727 (+ 2409us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:10:42.663730 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:10:42.663732 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:10:42.663753 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:10:42.665128 (+ 1375us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:10:42.665563 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:10:42.665566 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:10:42.665567 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:10:42.665635 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:10:42.665898 (+ 263us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:10:42.665900 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:10:42.665901 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:10:42.666056 (+ 155us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:10:42.666126 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0620 06:10:42.666540 (+ 414us) server_negotiation.cc:299] Negotiation successful
0620 06:10:42.666630 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260620 06:10:42.667536 29164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:10:42.655726 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.102.62:34917 (local address 127.11.102.3:47419)
0620 06:10:42.656348 (+ 622us) negotiation.cc:107] Waiting for socket to connect
0620 06:10:42.656389 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0620 06:10:42.657274 (+ 885us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:10:42.658014 (+ 740us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:10:42.658025 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:10:42.658495 (+ 470us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:10:42.659167 (+ 672us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:10:42.659187 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:10:42.659991 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:10:42.659993 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:10:42.660937 (+ 944us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:10:42.660946 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:10:42.661358 (+ 412us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:10:42.661361 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:10:42.661492 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:10:42.662034 (+ 542us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:10:42.662049 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:10:42.663528 (+ 1479us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:10:42.665224 (+ 1696us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:10:42.665228 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:10:42.665240 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:10:42.665488 (+ 248us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:10:42.665705 (+ 217us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:10:42.665707 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:10:42.665708 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:10:42.665825 (+ 117us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:10:42.666132 (+ 307us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:10:42.666137 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:10:42.666342 (+ 205us) client_negotiation.cc:769] Sending connection context
0620 06:10:42.666501 (+ 159us) client_negotiation.cc:240] Negotiation successful
0620 06:10:42.666732 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":441,"thread_start_us":113,"threads_started":1}
I20260620 06:10:42.668617 29162 heartbeater.cc:347] Connected to a master server at 127.11.102.62:34917
I20260620 06:10:42.668838 29162 heartbeater.cc:464] Registering TS with master...
I20260620 06:10:42.669293 29162 heartbeater.cc:511] Master 127.11.102.62:34917 requested a full tablet report, sending...
I20260620 06:10:42.670292 28703 ts_manager.cc:194] Registered new tserver with Master: 2ff12866e4c34bb1a57fdc24772fca03 (127.11.102.3:41009)
I20260620 06:10:42.670857 11672 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 06:10:42.670955 28703 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.102.3@KRBTEST.COM'} at 127.11.102.3:47419
I20260620 06:10:42.679242 28742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:10:42.672957 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40718 (local address 127.11.102.62:34917)
0620 06:10:42.673075 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0620 06:10:42.673079 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 06:10:42.673230 (+ 151us) server_negotiation.cc:407] Connection header received
0620 06:10:42.673332 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:10:42.673334 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:10:42.673366 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:10:42.673428 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:10:42.674035 (+ 607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:10:42.674509 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:10:42.675258 (+ 749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:10:42.675465 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:10:42.675539 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:10:42.676791 (+ 1252us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:10:42.676794 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:10:42.676796 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:10:42.676816 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:10:42.677922 (+ 1106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:10:42.678266 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:10:42.678268 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:10:42.678269 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:10:42.678362 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:10:42.678567 (+ 205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:10:42.678569 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:10:42.678570 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:10:42.678745 (+ 175us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:10:42.678839 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0620 06:10:42.679059 (+ 220us) server_negotiation.cc:299] Negotiation successful
0620 06:10:42.679139 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260620 06:10:42.681756 28703 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40718:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 06:10:42.684790 28703 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 06:10:42.698263 29176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:10:42.692909 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.102.3:41009 (local address 127.0.0.1:57324)
0620 06:10:42.693198 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0620 06:10:42.693212 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 06:10:42.693297 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:10:42.694002 (+ 705us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:10:42.694005 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:10:42.694024 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 06:10:42.694208 (+ 184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:10:42.694217 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:10:42.695252 (+ 1035us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:10:42.695257 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:10:42.696938 (+ 1681us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:10:42.696946 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:10:42.697727 (+ 781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:10:42.697731 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:10:42.697843 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:10:42.697880 (+ 37us) client_negotiation.cc:769] Sending connection context
0620 06:10:42.697983 (+ 103us) client_negotiation.cc:240] Negotiation successful
0620 06:10:42.698098 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":220,"mutex_wait_us":212,"thread_start_us":86,"threads_started":1}
I20260620 06:10:42.698482 29178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:10:42.693180 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53392 (local address 127.11.102.1:35423)
0620 06:10:42.693506 (+ 326us) server_negotiation.cc:206] Beginning negotiation
0620 06:10:42.693520 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 06:10:42.693544 (+ 24us) server_negotiation.cc:407] Connection header received
0620 06:10:42.693620 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:10:42.693626 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:10:42.693788 (+ 162us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:10:42.694139 (+ 351us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 06:10:42.694284 (+ 145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:10:42.695381 (+ 1097us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:10:42.697055 (+ 1674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:10:42.697839 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:10:42.697968 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:10:42.698104 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0620 06:10:42.698212 (+ 108us) server_negotiation.cc:299] Negotiation successful
0620 06:10:42.698320 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":243,"thread_start_us":72,"threads_started":1}
I20260620 06:10:42.698545 29174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:10:42.692184 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.102.2:34811 (local address 127.0.0.1:51626)
0620 06:10:42.692655 (+ 471us) negotiation.cc:107] Waiting for socket to connect
0620 06:10:42.692706 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0620 06:10:42.692838 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:10:42.693663 (+ 825us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:10:42.693669 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:10:42.693691 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 06:10:42.693849 (+ 158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:10:42.693860 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:10:42.695562 (+ 1702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:10:42.695564 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:10:42.696761 (+ 1197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:10:42.696775 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:10:42.697867 (+ 1092us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:10:42.697873 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:10:42.698212 (+ 339us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:10:42.698237 (+ 25us) client_negotiation.cc:769] Sending connection context
0620 06:10:42.698307 (+ 70us) client_negotiation.cc:240] Negotiation successful
0620 06:10:42.698371 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":297,"mutex_wait_us":240,"spinlock_wait_cycles":6656,"thread_start_us":72,"threads_started":1}
I20260620 06:10:42.698410 29173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:10:42.692293 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.102.1:35423 (local address 127.0.0.1:53392)
0620 06:10:42.692682 (+ 389us) negotiation.cc:107] Waiting for socket to connect
0620 06:10:42.692706 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0620 06:10:42.692838 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:10:42.693958 (+ 1120us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:10:42.693961 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:10:42.693978 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 06:10:42.694150 (+ 172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:10:42.694161 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:10:42.695505 (+ 1344us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:10:42.695508 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:10:42.696936 (+ 1428us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:10:42.696946 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:10:42.697970 (+ 1024us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:10:42.697977 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:10:42.698091 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:10:42.698111 (+ 20us) client_negotiation.cc:769] Sending connection context
0620 06:10:42.698182 (+ 71us) client_negotiation.cc:240] Negotiation successful
0620 06:10:42.698234 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":344,"mutex_wait_us":184,"spinlock_wait_cycles":2816,"thread_start_us":108,"threads_started":1}
I20260620 06:10:42.699045 29177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:10:42.693053 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57324 (local address 127.11.102.3:41009)
0620 06:10:42.693545 (+ 492us) server_negotiation.cc:206] Beginning negotiation
0620 06:10:42.693557 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 06:10:42.693577 (+ 20us) server_negotiation.cc:407] Connection header received
0620 06:10:42.693645 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:10:42.693651 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:10:42.693846 (+ 195us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:10:42.694339 (+ 493us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 06:10:42.694449 (+ 110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:10:42.695131 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:10:42.697075 (+ 1944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:10:42.697611 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:10:42.697727 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:10:42.697888 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0620 06:10:42.698762 (+ 874us) server_negotiation.cc:299] Negotiation successful
0620 06:10:42.698864 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":393,"thread_start_us":123,"threads_started":1}
I20260620 06:10:42.699122 29175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:10:42.692318 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51626 (local address 127.11.102.2:34811)
0620 06:10:42.693125 (+ 807us) server_negotiation.cc:206] Beginning negotiation
0620 06:10:42.693138 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 06:10:42.693162 (+ 24us) server_negotiation.cc:407] Connection header received
0620 06:10:42.693243 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:10:42.693251 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:10:42.693482 (+ 231us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:10:42.694014 (+ 532us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 06:10:42.694178 (+ 164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:10:42.695373 (+ 1195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:10:42.696907 (+ 1534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:10:42.697722 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:10:42.698037 (+ 315us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:10:42.698159 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0620 06:10:42.698856 (+ 697us) server_negotiation.cc:299] Negotiation successful
0620 06:10:42.698974 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":699,"thread_start_us":130,"threads_started":1}
I20260620 06:10:42.701514 29096 tablet_service.cc:1467] Processing CreateTablet for tablet bc7801435f89492da62636d5e60ac5e1 (DEFAULT_TABLE table=test-table [id=07f4854a57d7499195de073193160d10]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 06:10:42.701660 28828 tablet_service.cc:1467] Processing CreateTablet for tablet bc7801435f89492da62636d5e60ac5e1 (DEFAULT_TABLE table=test-table [id=07f4854a57d7499195de073193160d10]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 06:10:42.701660 28962 tablet_service.cc:1467] Processing CreateTablet for tablet bc7801435f89492da62636d5e60ac5e1 (DEFAULT_TABLE table=test-table [id=07f4854a57d7499195de073193160d10]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 06:10:42.702417 29096 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bc7801435f89492da62636d5e60ac5e1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 06:10:42.702617 28828 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bc7801435f89492da62636d5e60ac5e1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 06:10:42.702617 28962 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bc7801435f89492da62636d5e60ac5e1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 06:10:42.708793 29179 tablet_bootstrap.cc:492] T bc7801435f89492da62636d5e60ac5e1 P 2ff12866e4c34bb1a57fdc24772fca03: Bootstrap starting.
I20260620 06:10:42.709298 29180 tablet_bootstrap.cc:492] T bc7801435f89492da62636d5e60ac5e1 P f75880b31b994309ad340701e0fc9237: Bootstrap starting.
I20260620 06:10:42.709682 29181 tablet_bootstrap.cc:492] T bc7801435f89492da62636d5e60ac5e1 P 7040f5853eba4bf6967fda1b4952f45e: Bootstrap starting.
I20260620 06:10:42.711560 29179 tablet_bootstrap.cc:654] T bc7801435f89492da62636d5e60ac5e1 P 2ff12866e4c34bb1a57fdc24772fca03: Neither blocks nor log segments found. Creating new log.
I20260620 06:10:42.711589 29180 tablet_bootstrap.cc:654] T bc7801435f89492da62636d5e60ac5e1 P f75880b31b994309ad340701e0fc9237: Neither blocks nor log segments found. Creating new log.
I20260620 06:10:42.712080 29181 tablet_bootstrap.cc:654] T bc7801435f89492da62636d5e60ac5e1 P 7040f5853eba4bf6967fda1b4952f45e: Neither blocks nor log segments found. Creating new log.
I20260620 06:10:42.712648 29180 log.cc:824] T bc7801435f89492da62636d5e60ac5e1 P f75880b31b994309ad340701e0fc9237: Log is configured to *not* fsync() on all Append() calls
I20260620 06:10:42.712802 29179 log.cc:824] T bc7801435f89492da62636d5e60ac5e1 P 2ff12866e4c34bb1a57fdc24772fca03: Log is configured to *not* fsync() on all Append() calls
I20260620 06:10:42.712802 29181 log.cc:824] T bc7801435f89492da62636d5e60ac5e1 P 7040f5853eba4bf6967fda1b4952f45e: Log is configured to *not* fsync() on all Append() calls
I20260620 06:10:42.714583 29179 tablet_bootstrap.cc:492] T bc7801435f89492da62636d5e60ac5e1 P 2ff12866e4c34bb1a57fdc24772fca03: No bootstrap required, opened a new log
I20260620 06:10:42.714583 29181 tablet_bootstrap.cc:492] T bc7801435f89492da62636d5e60ac5e1 P 7040f5853eba4bf6967fda1b4952f45e: No bootstrap required, opened a new log
I20260620 06:10:42.714592 29180 tablet_bootstrap.cc:492] T bc7801435f89492da62636d5e60ac5e1 P f75880b31b994309ad340701e0fc9237: No bootstrap required, opened a new log
I20260620 06:10:42.714767 29179 ts_tablet_manager.cc:1397] T bc7801435f89492da62636d5e60ac5e1 P 2ff12866e4c34bb1a57fdc24772fca03: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.003s
I20260620 06:10:42.714767 29181 ts_tablet_manager.cc:1397] T bc7801435f89492da62636d5e60ac5e1 P 7040f5853eba4bf6967fda1b4952f45e: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260620 06:10:42.714782 29180 ts_tablet_manager.cc:1397] T bc7801435f89492da62636d5e60ac5e1 P f75880b31b994309ad340701e0fc9237: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.000s
I20260620 06:10:42.717499 29180 raft_consensus.cc:348] T bc7801435f89492da62636d5e60ac5e1 P f75880b31b994309ad340701e0fc9237 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7040f5853eba4bf6967fda1b4952f45e" member_type: VOTER last_known_addr { host: "127.11.102.2" port: 34811 } } peers { permanent_uuid: "f75880b31b994309ad340701e0fc9237" member_type: VOTER last_known_addr { host: "127.11.102.1" port: 35423 } } peers { permanent_uuid: "2ff12866e4c34bb1a57fdc24772fca03" member_type: VOTER last_known_addr { host: "127.11.102.3" port: 41009 } }
I20260620 06:10:42.717717 29180 raft_consensus.cc:374] T bc7801435f89492da62636d5e60ac5e1 P f75880b31b994309ad340701e0fc9237 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 06:10:42.717789 29180 raft_consensus.cc:729] T bc7801435f89492da62636d5e60ac5e1 P f75880b31b994309ad340701e0fc9237 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f75880b31b994309ad340701e0fc9237, State: Initialized, Role: FOLLOWER
I20260620 06:10:42.718226 29180 consensus_queue.cc:260] T bc7801435f89492da62636d5e60ac5e1 P f75880b31b994309ad340701e0fc9237 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7040f5853eba4bf6967fda1b4952f45e" member_type: VOTER last_known_addr { host: "127.11.102.2" port: 34811 } } peers { permanent_uuid: "f75880b31b994309ad340701e0fc9237" member_type: VOTER last_known_addr { host: "127.11.102.1" port: 35423 } } peers { permanent_uuid: "2ff12866e4c34bb1a57fdc24772fca03" member_type: VOTER last_known_addr { host: "127.11.102.3" port: 41009 } }
I20260620 06:10:42.718490 29179 raft_consensus.cc:348] T bc7801435f89492da62636d5e60ac5e1 P 2ff12866e4c34bb1a57fdc24772fca03 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7040f5853eba4bf6967fda1b4952f45e" member_type: VOTER last_known_addr { host: "127.11.102.2" port: 34811 } } peers { permanent_uuid: "f75880b31b994309ad340701e0fc9237" member_type: VOTER last_known_addr { host: "127.11.102.1" port: 35423 } } peers { permanent_uuid: "2ff12866e4c34bb1a57fdc24772fca03" member_type: VOTER last_known_addr { host: "127.11.102.3" port: 41009 } }
I20260620 06:10:42.718487 29181 raft_consensus.cc:348] T bc7801435f89492da62636d5e60ac5e1 P 7040f5853eba4bf6967fda1b4952f45e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7040f5853eba4bf6967fda1b4952f45e" member_type: VOTER last_known_addr { host: "127.11.102.2" port: 34811 } } peers { permanent_uuid: "f75880b31b994309ad340701e0fc9237" member_type: VOTER last_known_addr { host: "127.11.102.1" port: 35423 } } peers { permanent_uuid: "2ff12866e4c34bb1a57fdc24772fca03" member_type: VOTER last_known_addr { host: "127.11.102.3" port: 41009 } }
I20260620 06:10:42.718740 29181 raft_consensus.cc:374] T bc7801435f89492da62636d5e60ac5e1 P 7040f5853eba4bf6967fda1b4952f45e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 06:10:42.718801 29179 raft_consensus.cc:374] T bc7801435f89492da62636d5e60ac5e1 P 2ff12866e4c34bb1a57fdc24772fca03 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 06:10:42.718834 29181 raft_consensus.cc:729] T bc7801435f89492da62636d5e60ac5e1 P 7040f5853eba4bf6967fda1b4952f45e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7040f5853eba4bf6967fda1b4952f45e, State: Initialized, Role: FOLLOWER
I20260620 06:10:42.718856 28894 heartbeater.cc:503] Master 127.11.102.62:34917 was elected leader, sending a full tablet report...
I20260620 06:10:42.718891 29179 raft_consensus.cc:729] T bc7801435f89492da62636d5e60ac5e1 P 2ff12866e4c34bb1a57fdc24772fca03 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2ff12866e4c34bb1a57fdc24772fca03, State: Initialized, Role: FOLLOWER
I20260620 06:10:42.719202 29180 ts_tablet_manager.cc:1428] T bc7801435f89492da62636d5e60ac5e1 P f75880b31b994309ad340701e0fc9237: Time spent starting tablet: real 0.004s user 0.005s sys 0.000s
I20260620 06:10:42.719369 29181 consensus_queue.cc:260] T bc7801435f89492da62636d5e60ac5e1 P 7040f5853eba4bf6967fda1b4952f45e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7040f5853eba4bf6967fda1b4952f45e" member_type: VOTER last_known_addr { host: "127.11.102.2" port: 34811 } } peers { permanent_uuid: "f75880b31b994309ad340701e0fc9237" member_type: VOTER last_known_addr { host: "127.11.102.1" port: 35423 } } peers { permanent_uuid: "2ff12866e4c34bb1a57fdc24772fca03" member_type: VOTER last_known_addr { host: "127.11.102.3" port: 41009 } }
I20260620 06:10:42.719369 29179 consensus_queue.cc:260] T bc7801435f89492da62636d5e60ac5e1 P 2ff12866e4c34bb1a57fdc24772fca03 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7040f5853eba4bf6967fda1b4952f45e" member_type: VOTER last_known_addr { host: "127.11.102.2" port: 34811 } } peers { permanent_uuid: "f75880b31b994309ad340701e0fc9237" member_type: VOTER last_known_addr { host: "127.11.102.1" port: 35423 } } peers { permanent_uuid: "2ff12866e4c34bb1a57fdc24772fca03" member_type: VOTER last_known_addr { host: "127.11.102.3" port: 41009 } }
I20260620 06:10:42.720259 29162 heartbeater.cc:503] Master 127.11.102.62:34917 was elected leader, sending a full tablet report...
I20260620 06:10:42.720288 29181 ts_tablet_manager.cc:1428] T bc7801435f89492da62636d5e60ac5e1 P 7040f5853eba4bf6967fda1b4952f45e: Time spent starting tablet: real 0.005s user 0.006s sys 0.001s
I20260620 06:10:42.720288 29179 ts_tablet_manager.cc:1428] T bc7801435f89492da62636d5e60ac5e1 P 2ff12866e4c34bb1a57fdc24772fca03: Time spent starting tablet: real 0.005s user 0.004s sys 0.002s
I20260620 06:10:42.720587 29028 heartbeater.cc:503] Master 127.11.102.62:34917 was elected leader, sending a full tablet report...
I20260620 06:10:42.807881 29187 raft_consensus.cc:482] T bc7801435f89492da62636d5e60ac5e1 P 2ff12866e4c34bb1a57fdc24772fca03 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 06:10:42.808089 29187 raft_consensus.cc:504] T bc7801435f89492da62636d5e60ac5e1 P 2ff12866e4c34bb1a57fdc24772fca03 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7040f5853eba4bf6967fda1b4952f45e" member_type: VOTER last_known_addr { host: "127.11.102.2" port: 34811 } } peers { permanent_uuid: "f75880b31b994309ad340701e0fc9237" member_type: VOTER last_known_addr { host: "127.11.102.1" port: 35423 } } peers { permanent_uuid: "2ff12866e4c34bb1a57fdc24772fca03" member_type: VOTER last_known_addr { host: "127.11.102.3" port: 41009 } }
I20260620 06:10:42.809230 29187 leader_election.cc:288] T bc7801435f89492da62636d5e60ac5e1 P 2ff12866e4c34bb1a57fdc24772fca03 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7040f5853eba4bf6967fda1b4952f45e (127.11.102.2:34811), f75880b31b994309ad340701e0fc9237 (127.11.102.1:35423)
I20260620 06:10:42.813177 29178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:10:42.809691 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.102.3:38035 (local address 127.11.102.1:35423)
0620 06:10:42.809848 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0620 06:10:42.809851 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 06:10:42.809866 (+ 15us) server_negotiation.cc:407] Connection header received
0620 06:10:42.809916 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:10:42.809918 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:10:42.809958 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:10:42.810024 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 06:10:42.810338 (+ 314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:10:42.811041 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:10:42.812244 (+ 1203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:10:42.812642 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:10:42.812764 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:10:42.812814 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0620 06:10:42.812996 (+ 182us) server_negotiation.cc:299] Negotiation successful
0620 06:10:42.813066 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260620 06:10:42.813179 29164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:10:42.809516 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.102.1:35423 (local address 127.11.102.3:38035)
0620 06:10:42.809644 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0620 06:10:42.809661 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 06:10:42.809756 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:10:42.810079 (+ 323us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:10:42.810082 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:10:42.810101 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 06:10:42.810228 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:10:42.810234 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:10:42.811119 (+ 885us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:10:42.811125 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:10:42.812119 (+ 994us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:10:42.812129 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:10:42.812754 (+ 625us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:10:42.812758 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:10:42.812870 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:10:42.812895 (+ 25us) client_negotiation.cc:769] Sending connection context
0620 06:10:42.812980 (+ 85us) client_negotiation.cc:240] Negotiation successful
0620 06:10:42.813048 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":36}
I20260620 06:10:42.813796 28848 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc7801435f89492da62636d5e60ac5e1" candidate_uuid: "2ff12866e4c34bb1a57fdc24772fca03" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f75880b31b994309ad340701e0fc9237" is_pre_election: true
I20260620 06:10:42.814103 28848 raft_consensus.cc:2454] T bc7801435f89492da62636d5e60ac5e1 P f75880b31b994309ad340701e0fc9237 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2ff12866e4c34bb1a57fdc24772fca03 in term 0.
I20260620 06:10:42.814453 29188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:10:42.809513 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.102.2:34811 (local address 127.11.102.3:48371)
0620 06:10:42.810216 (+ 703us) negotiation.cc:107] Waiting for socket to connect
0620 06:10:42.810227 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 06:10:42.810353 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:10:42.810821 (+ 468us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:10:42.810824 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:10:42.810869 (+ 45us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 06:10:42.810976 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:10:42.810982 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:10:42.811955 (+ 973us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:10:42.811961 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:10:42.812738 (+ 777us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:10:42.812746 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:10:42.813481 (+ 735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:10:42.813485 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:10:42.814107 (+ 622us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:10:42.814144 (+ 37us) client_negotiation.cc:769] Sending connection context
0620 06:10:42.814226 (+ 82us) client_negotiation.cc:240] Negotiation successful
0620 06:10:42.814288 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":646,"mutex_wait_us":331,"thread_start_us":151,"threads_started":1}
I20260620 06:10:42.814529 29175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:10:42.809558 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.102.3:48371 (local address 127.11.102.2:34811)
0620 06:10:42.809703 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0620 06:10:42.809706 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 06:10:42.810315 (+ 609us) server_negotiation.cc:407] Connection header received
0620 06:10:42.810484 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:10:42.810488 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:10:42.810541 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:10:42.810626 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 06:10:42.811105 (+ 479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:10:42.811847 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:10:42.812871 (+ 1024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:10:42.813358 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:10:42.813481 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:10:42.813526 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0620 06:10:42.814263 (+ 737us) server_negotiation.cc:299] Negotiation successful
0620 06:10:42.814388 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260620 06:10:42.814610 29049 leader_election.cc:302] T bc7801435f89492da62636d5e60ac5e1 P 2ff12866e4c34bb1a57fdc24772fca03 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2ff12866e4c34bb1a57fdc24772fca03, f75880b31b994309ad340701e0fc9237; no voters:
I20260620 06:10:42.814901 29187 raft_consensus.cc:2781] T bc7801435f89492da62636d5e60ac5e1 P 2ff12866e4c34bb1a57fdc24772fca03 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 06:10:42.814997 29187 raft_consensus.cc:482] T bc7801435f89492da62636d5e60ac5e1 P 2ff12866e4c34bb1a57fdc24772fca03 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 06:10:42.815053 29187 raft_consensus.cc:3037] T bc7801435f89492da62636d5e60ac5e1 P 2ff12866e4c34bb1a57fdc24772fca03 [term 0 FOLLOWER]: Advancing to term 1
I20260620 06:10:42.815124 28982 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc7801435f89492da62636d5e60ac5e1" candidate_uuid: "2ff12866e4c34bb1a57fdc24772fca03" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7040f5853eba4bf6967fda1b4952f45e" is_pre_election: true
I20260620 06:10:42.815404 28982 raft_consensus.cc:2454] T bc7801435f89492da62636d5e60ac5e1 P 7040f5853eba4bf6967fda1b4952f45e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2ff12866e4c34bb1a57fdc24772fca03 in term 0.
I20260620 06:10:42.816303 29187 raft_consensus.cc:504] T bc7801435f89492da62636d5e60ac5e1 P 2ff12866e4c34bb1a57fdc24772fca03 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7040f5853eba4bf6967fda1b4952f45e" member_type: VOTER last_known_addr { host: "127.11.102.2" port: 34811 } } peers { permanent_uuid: "f75880b31b994309ad340701e0fc9237" member_type: VOTER last_known_addr { host: "127.11.102.1" port: 35423 } } peers { permanent_uuid: "2ff12866e4c34bb1a57fdc24772fca03" member_type: VOTER last_known_addr { host: "127.11.102.3" port: 41009 } }
I20260620 06:10:42.816640 29187 leader_election.cc:288] T bc7801435f89492da62636d5e60ac5e1 P 2ff12866e4c34bb1a57fdc24772fca03 [CANDIDATE]: Term 1 election: Requested vote from peers 7040f5853eba4bf6967fda1b4952f45e (127.11.102.2:34811), f75880b31b994309ad340701e0fc9237 (127.11.102.1:35423)
I20260620 06:10:42.816993 28848 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc7801435f89492da62636d5e60ac5e1" candidate_uuid: "2ff12866e4c34bb1a57fdc24772fca03" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f75880b31b994309ad340701e0fc9237"
I20260620 06:10:42.817001 28982 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc7801435f89492da62636d5e60ac5e1" candidate_uuid: "2ff12866e4c34bb1a57fdc24772fca03" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7040f5853eba4bf6967fda1b4952f45e"
I20260620 06:10:42.817119 28982 raft_consensus.cc:3037] T bc7801435f89492da62636d5e60ac5e1 P 7040f5853eba4bf6967fda1b4952f45e [term 0 FOLLOWER]: Advancing to term 1
I20260620 06:10:42.817118 28848 raft_consensus.cc:3037] T bc7801435f89492da62636d5e60ac5e1 P f75880b31b994309ad340701e0fc9237 [term 0 FOLLOWER]: Advancing to term 1
I20260620 06:10:42.818365 28982 raft_consensus.cc:2454] T bc7801435f89492da62636d5e60ac5e1 P 7040f5853eba4bf6967fda1b4952f45e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2ff12866e4c34bb1a57fdc24772fca03 in term 1.
I20260620 06:10:42.818363 28848 raft_consensus.cc:2454] T bc7801435f89492da62636d5e60ac5e1 P f75880b31b994309ad340701e0fc9237 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2ff12866e4c34bb1a57fdc24772fca03 in term 1.
I20260620 06:10:42.818648 29049 leader_election.cc:302] T bc7801435f89492da62636d5e60ac5e1 P 2ff12866e4c34bb1a57fdc24772fca03 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 2ff12866e4c34bb1a57fdc24772fca03, 7040f5853eba4bf6967fda1b4952f45e, f75880b31b994309ad340701e0fc9237; no voters:
I20260620 06:10:42.818842 29187 raft_consensus.cc:2781] T bc7801435f89492da62636d5e60ac5e1 P 2ff12866e4c34bb1a57fdc24772fca03 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 06:10:42.819032 29187 raft_consensus.cc:686] T bc7801435f89492da62636d5e60ac5e1 P 2ff12866e4c34bb1a57fdc24772fca03 [term 1 LEADER]: Becoming Leader. State: Replica: 2ff12866e4c34bb1a57fdc24772fca03, State: Running, Role: LEADER
I20260620 06:10:42.819341 29187 consensus_queue.cc:237] T bc7801435f89492da62636d5e60ac5e1 P 2ff12866e4c34bb1a57fdc24772fca03 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7040f5853eba4bf6967fda1b4952f45e" member_type: VOTER last_known_addr { host: "127.11.102.2" port: 34811 } } peers { permanent_uuid: "f75880b31b994309ad340701e0fc9237" member_type: VOTER last_known_addr { host: "127.11.102.1" port: 35423 } } peers { permanent_uuid: "2ff12866e4c34bb1a57fdc24772fca03" member_type: VOTER last_known_addr { host: "127.11.102.3" port: 41009 } }
I20260620 06:10:42.822497 28701 catalog_manager.cc:5495] T bc7801435f89492da62636d5e60ac5e1 P 2ff12866e4c34bb1a57fdc24772fca03 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2ff12866e4c34bb1a57fdc24772fca03 (127.11.102.3). New cstate: current_term: 1 leader_uuid: "2ff12866e4c34bb1a57fdc24772fca03" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7040f5853eba4bf6967fda1b4952f45e" member_type: VOTER last_known_addr { host: "127.11.102.2" port: 34811 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f75880b31b994309ad340701e0fc9237" member_type: VOTER last_known_addr { host: "127.11.102.1" port: 35423 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2ff12866e4c34bb1a57fdc24772fca03" member_type: VOTER last_known_addr { host: "127.11.102.3" port: 41009 } health_report { overall_health: HEALTHY } } }
I20260620 06:10:42.848701 29177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:10:42.845458 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57336 (local address 127.11.102.3:41009)
0620 06:10:42.845586 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0620 06:10:42.845589 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 06:10:42.845603 (+ 14us) server_negotiation.cc:407] Connection header received
0620 06:10:42.845749 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:10:42.845752 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:10:42.845804 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:10:42.845904 (+ 100us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 06:10:42.846108 (+ 204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:10:42.846591 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:10:42.847381 (+ 790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:10:42.847586 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:10:42.847678 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:10:42.848005 (+ 327us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 06:10:42.848403 (+ 398us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 06:10:42.848497 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0620 06:10:42.848553 (+ 56us) server_negotiation.cc:299] Negotiation successful
0620 06:10:42.848599 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260620 06:10:42.854187 28848 raft_consensus.cc:1261] T bc7801435f89492da62636d5e60ac5e1 P f75880b31b994309ad340701e0fc9237 [term 1 FOLLOWER]: Refusing update from remote peer 2ff12866e4c34bb1a57fdc24772fca03: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 06:10:42.854188 28982 raft_consensus.cc:1261] T bc7801435f89492da62636d5e60ac5e1 P 7040f5853eba4bf6967fda1b4952f45e [term 1 FOLLOWER]: Refusing update from remote peer 2ff12866e4c34bb1a57fdc24772fca03: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 06:10:42.854873 29187 consensus_queue.cc:1035] T bc7801435f89492da62636d5e60ac5e1 P 2ff12866e4c34bb1a57fdc24772fca03 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f75880b31b994309ad340701e0fc9237" member_type: VOTER last_known_addr { host: "127.11.102.1" port: 35423 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 06:10:42.855064 29189 consensus_queue.cc:1035] T bc7801435f89492da62636d5e60ac5e1 P 2ff12866e4c34bb1a57fdc24772fca03 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7040f5853eba4bf6967fda1b4952f45e" member_type: VOTER last_known_addr { host: "127.11.102.2" port: 34811 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 06:10:42.862210 29192 mvcc.cc:205] Tried to move back new op lower bound from 7298809212324048896 to 7298809212190892032. Current Snapshot: MvccSnapshot[applied={T|T < 7298809212324048896}]
I20260620 06:10:42.862824 29194 mvcc.cc:205] Tried to move back new op lower bound from 7298809212324048896 to 7298809212190892032. Current Snapshot: MvccSnapshot[applied={T|T < 7298809212324048896}]
I20260620 06:10:42.865186 29193 mvcc.cc:205] Tried to move back new op lower bound from 7298809212324048896 to 7298809212190892032. Current Snapshot: MvccSnapshot[applied={T|T < 7298809212324048896 or (T in {7298809212324048896})}]
I20260620 06:10:42.872308 28700 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40718:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 06:10:42.872457 28700 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40718:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 06:10:42.874791 28700 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 7fc9e7ca1b284980816a651a762ff0ff: Sending DeleteTablet for 3 replicas of tablet bc7801435f89492da62636d5e60ac5e1
I20260620 06:10:42.875517 28962 tablet_service.cc:1514] Processing DeleteTablet for tablet bc7801435f89492da62636d5e60ac5e1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 06:10:42 UTC) from {username='kudu', principal='kudu/127.11.102.62@KRBTEST.COM'} at 127.0.0.1:51626
I20260620 06:10:42.875633 29096 tablet_service.cc:1514] Processing DeleteTablet for tablet bc7801435f89492da62636d5e60ac5e1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 06:10:42 UTC) from {username='kudu', principal='kudu/127.11.102.62@KRBTEST.COM'} at 127.0.0.1:57324
I20260620 06:10:42.875825 28828 tablet_service.cc:1514] Processing DeleteTablet for tablet bc7801435f89492da62636d5e60ac5e1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 06:10:42 UTC) from {username='kudu', principal='kudu/127.11.102.62@KRBTEST.COM'} at 127.0.0.1:53392
I20260620 06:10:42.876060 29202 tablet_replica.cc:330] T bc7801435f89492da62636d5e60ac5e1 P 7040f5853eba4bf6967fda1b4952f45e: stopping tablet replica
I20260620 06:10:42.876075 29203 tablet_replica.cc:330] T bc7801435f89492da62636d5e60ac5e1 P 2ff12866e4c34bb1a57fdc24772fca03: stopping tablet replica
I20260620 06:10:42.876353 29203 raft_consensus.cc:2229] T bc7801435f89492da62636d5e60ac5e1 P 2ff12866e4c34bb1a57fdc24772fca03 [term 1 LEADER]: Raft consensus shutting down.
I20260620 06:10:42.876369 29204 tablet_replica.cc:330] T bc7801435f89492da62636d5e60ac5e1 P f75880b31b994309ad340701e0fc9237: stopping tablet replica
I20260620 06:10:42.876379 29202 raft_consensus.cc:2229] T bc7801435f89492da62636d5e60ac5e1 P 7040f5853eba4bf6967fda1b4952f45e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 06:10:42.876641 29204 raft_consensus.cc:2229] T bc7801435f89492da62636d5e60ac5e1 P f75880b31b994309ad340701e0fc9237 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 06:10:42.876667 29202 raft_consensus.cc:2258] T bc7801435f89492da62636d5e60ac5e1 P 7040f5853eba4bf6967fda1b4952f45e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 06:10:42.876780 11672 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9HMp43/build/debug/bin/kudu with pid 28765
I20260620 06:10:42.876861 29203 raft_consensus.cc:2258] T bc7801435f89492da62636d5e60ac5e1 P 2ff12866e4c34bb1a57fdc24772fca03 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 06:10:42.876983 29204 raft_consensus.cc:2258] T bc7801435f89492da62636d5e60ac5e1 P f75880b31b994309ad340701e0fc9237 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 06:10:42.878218 29203 ts_tablet_manager.cc:1905] T bc7801435f89492da62636d5e60ac5e1 P 2ff12866e4c34bb1a57fdc24772fca03: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 06:10:42.878974 29202 ts_tablet_manager.cc:1905] T bc7801435f89492da62636d5e60ac5e1 P 7040f5853eba4bf6967fda1b4952f45e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 06:10:42.880782 29203 ts_tablet_manager.cc:1918] T bc7801435f89492da62636d5e60ac5e1 P 2ff12866e4c34bb1a57fdc24772fca03: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 06:10:42.880920 29203 log.cc:1196] T bc7801435f89492da62636d5e60ac5e1 P 2ff12866e4c34bb1a57fdc24772fca03: Deleting WAL directory at /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781935603120653-11672-0/minicluster-data/ts-2/wal/wals/bc7801435f89492da62636d5e60ac5e1
I20260620 06:10:42.881284 29203 ts_tablet_manager.cc:1939] T bc7801435f89492da62636d5e60ac5e1 P 2ff12866e4c34bb1a57fdc24772fca03: Deleting consensus metadata
I20260620 06:10:42.881989 29202 ts_tablet_manager.cc:1918] T bc7801435f89492da62636d5e60ac5e1 P 7040f5853eba4bf6967fda1b4952f45e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 06:10:42.882114 29202 log.cc:1196] T bc7801435f89492da62636d5e60ac5e1 P 7040f5853eba4bf6967fda1b4952f45e: Deleting WAL directory at /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781935603120653-11672-0/minicluster-data/ts-1/wal/wals/bc7801435f89492da62636d5e60ac5e1
I20260620 06:10:42.882468 29202 ts_tablet_manager.cc:1939] T bc7801435f89492da62636d5e60ac5e1 P 7040f5853eba4bf6967fda1b4952f45e: Deleting consensus metadata
I20260620 06:10:42.883656 28689 catalog_manager.cc:4841] TS 7040f5853eba4bf6967fda1b4952f45e (127.11.102.2:34811): tablet bc7801435f89492da62636d5e60ac5e1 (table test-table [id=07f4854a57d7499195de073193160d10]) successfully deleted
W20260620 06:10:42.885259 28687 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.11.102.1:35423 (error 108)
I20260620 06:10:42.885702 28687 catalog_manager.cc:4841] TS 2ff12866e4c34bb1a57fdc24772fca03 (127.11.102.3:41009): tablet bc7801435f89492da62636d5e60ac5e1 (table test-table [id=07f4854a57d7499195de073193160d10]) successfully deleted
I20260620 06:10:42.886250 29173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:10:42.885983 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.102.1:35423 (local address 127.0.0.1:53406)
0620 06:10:42.886112 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0620 06:10:42.886181 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.102.1:35423: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260620 06:10:42.886255 11672 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9HMp43/build/debug/bin/kudu with pid 28899
W20260620 06:10:42.887192 28687 catalog_manager.cc:4584] TS f75880b31b994309ad340701e0fc9237 (127.11.102.1:35423): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet bc7801435f89492da62636d5e60ac5e1: Network error: Client connection negotiation failed: client connection to 127.11.102.1:35423: connect: Connection refused (error 111)
I20260620 06:10:42.893528 11672 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9HMp43/build/debug/bin/kudu with pid 29033
I20260620 06:10:42.899176 11672 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9HMp43/build/debug/bin/kudu with pid 28671
2026-06-20T06:10:42Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (2318 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-06-20T06:10:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T06:10:42Z Disabled control of system clock
I20260620 06:10:42.929219 11672 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9HMp43/build/debug/bin/kudu
/tmp/dist-test-task9HMp43/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781935603120653-11672-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781935603120653-11672-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781935603120653-11672-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781935603120653-11672-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.102.62:32993
--webserver_interface=127.11.102.62
--webserver_port=0
--builtin_ntp_servers=127.11.102.20:33837
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.102.62:32993
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260620 06:10:43.037565 29209 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:10:43.037814 29209 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:10:43.037848 29209 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:10:43.041755 29209 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 06:10:43.041806 29209 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:10:43.041822 29209 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:10:43.041841 29209 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 06:10:43.041855 29209 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 06:10:43.046497 29209 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.102.20:33837
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781935603120653-11672-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781935603120653-11672-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.102.62:32993
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.102.62:32993
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781935603120653-11672-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.102.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29209
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781935603120653-11672-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 05:28:48 UTC on ccycloud.quasar-fixhms.root.comops.site
build id 4457020
I20260620 06:10:43.047451 29209 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:10:43.048815 29209 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 06:10:43.056942 29217 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:10:43.057102 29209 server_base.cc:1034] running on GCE node
W20260620 06:10:43.056929 29215 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:10:43.056929 29214 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:10:43.057852 29209 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:10:43.058926 29209 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:10:43.060125 29209 hybrid_clock.cc:648] HybridClock initialized: now 1781935843060110 us; error 28 us; skew 500 ppm
I20260620 06:10:43.062002 29209 webserver.cc:460] Webserver started at http://127.11.102.62:41065/ using document root <none> and password file <none>
I20260620 06:10:43.062582 29209 fs_manager.cc:359] Metadata directory not provided
I20260620 06:10:43.062685 29209 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:10:43.062907 29209 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 06:10:43.064962 29209 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781935603120653-11672-0/minicluster-data/master-0/data/instance:
uuid: "e527a90723fc4a0288ad21cb7d0379cc"
format_stamp: "Formatted at 2026-06-20 06:10:43 on dist-test-slave-9hm5"
server_key: "0867058982b045b6be4d827785b48b29"
server_key_iv: "31509566995980492923c95794514968"
server_key_version: "encryptionkey@0"
I20260620 06:10:43.065506 29209 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781935603120653-11672-0/minicluster-data/master-0/wal/instance:
uuid: "e527a90723fc4a0288ad21cb7d0379cc"
format_stamp: "Formatted at 2026-06-20 06:10:43 on dist-test-slave-9hm5"
server_key: "0867058982b045b6be4d827785b48b29"
server_key_iv: "31509566995980492923c95794514968"
server_key_version: "encryptionkey@0"
I20260620 06:10:43.069180 29209 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260620 06:10:43.071496 29223 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:10:43.072719 29209 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 06:10:43.072863 29209 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781935603120653-11672-0/minicluster-data/master-0/data,/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781935603120653-11672-0/minicluster-data/master-0/wal
uuid: "e527a90723fc4a0288ad21cb7d0379cc"
format_stamp: "Formatted at 2026-06-20 06:10:43 on dist-test-slave-9hm5"
server_key: "0867058982b045b6be4d827785b48b29"
server_key_iv: "31509566995980492923c95794514968"
server_key_version: "encryptionkey@0"
I20260620 06:10:43.072970 29209 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781935603120653-11672-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781935603120653-11672-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781935603120653-11672-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:10:43.097396 29209 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:10:43.098114 29209 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:10:43.098318 29209 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:10:43.105229 29209 rpc_server.cc:307] RPC server started. Bound to: 127.11.102.62:32993
I20260620 06:10:43.105238 29275 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.102.62:32993 every 8 connection(s)
I20260620 06:10:43.106149 29209 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781935603120653-11672-0/minicluster-data/master-0/data/info.pb
I20260620 06:10:43.109239 29276 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 06:10:43.114552 29276 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e527a90723fc4a0288ad21cb7d0379cc: Bootstrap starting.
I20260620 06:10:43.114725 11672 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9HMp43/build/debug/bin/kudu as pid 29209
I20260620 06:10:43.114864 11672 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781935603120653-11672-0/minicluster-data/master-0/wal/instance
I20260620 06:10:43.115154 11672 external_mini_cluster.cc:1428] Setting key 224d2fa3a89a6f9c9467a85daf9ea103
I20260620 06:10:43.117421 29276 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e527a90723fc4a0288ad21cb7d0379cc: Neither blocks nor log segments found. Creating new log.
I20260620 06:10:43.118160 29276 log.cc:824] T 00000000000000000000000000000000 P e527a90723fc4a0288ad21cb7d0379cc: Log is configured to *not* fsync() on all Append() calls
I20260620 06:10:43.120210 29276 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e527a90723fc4a0288ad21cb7d0379cc: No bootstrap required, opened a new log
I20260620 06:10:43.124532 29276 raft_consensus.cc:348] T 00000000000000000000000000000000 P e527a90723fc4a0288ad21cb7d0379cc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e527a90723fc4a0288ad21cb7d0379cc" member_type: VOTER last_known_addr { host: "127.11.102.62" port: 32993 } }
I20260620 06:10:43.124744 29276 raft_consensus.cc:374] T 00000000000000000000000000000000 P e527a90723fc4a0288ad21cb7d0379cc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 06:10:43.124720 29279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:10:43.116591 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36104 (local address 127.11.102.62:32993)
0620 06:10:43.117258 (+ 667us) server_negotiation.cc:206] Beginning negotiation
0620 06:10:43.117272 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 06:10:43.117307 (+ 35us) server_negotiation.cc:407] Connection header received
0620 06:10:43.117992 (+ 685us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:10:43.118010 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:10:43.118449 (+ 439us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:10:43.118820 (+ 371us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:10:43.119212 (+ 392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:10:43.120242 (+ 1030us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:10:43.121489 (+ 1247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:10:43.121957 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:10:43.122077 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:10:43.122924 (+ 847us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:10:43.122929 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:10:43.122941 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 06:10:43.123324 (+ 383us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:10:43.123379 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 06:10:43.123401 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:10:43.123479 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0620 06:10:43.123718 (+ 239us) server_negotiation.cc:299] Negotiation successful
0620 06:10:43.124009 (+ 291us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":308,"thread_start_us":140,"threads_started":1}
I20260620 06:10:43.124786 29276 raft_consensus.cc:729] T 00000000000000000000000000000000 P e527a90723fc4a0288ad21cb7d0379cc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e527a90723fc4a0288ad21cb7d0379cc, State: Initialized, Role: FOLLOWER
I20260620 06:10:43.125312 29276 consensus_queue.cc:260] T 00000000000000000000000000000000 P e527a90723fc4a0288ad21cb7d0379cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e527a90723fc4a0288ad21cb7d0379cc" member_type: VOTER last_known_addr { host: "127.11.102.62" port: 32993 } }
I20260620 06:10:43.125471 29276 raft_consensus.cc:388] T 00000000000000000000000000000000 P e527a90723fc4a0288ad21cb7d0379cc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 06:10:43.125537 29276 raft_consensus.cc:482] T 00000000000000000000000000000000 P e527a90723fc4a0288ad21cb7d0379cc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 06:10:43.125659 29276 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e527a90723fc4a0288ad21cb7d0379cc [term 0 FOLLOWER]: Advancing to term 1
I20260620 06:10:43.126524 29276 raft_consensus.cc:504] T 00000000000000000000000000000000 P e527a90723fc4a0288ad21cb7d0379cc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e527a90723fc4a0288ad21cb7d0379cc" member_type: VOTER last_known_addr { host: "127.11.102.62" port: 32993 } }
I20260620 06:10:43.126886 29276 leader_election.cc:302] T 00000000000000000000000000000000 P e527a90723fc4a0288ad21cb7d0379cc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e527a90723fc4a0288ad21cb7d0379cc; no voters:
I20260620 06:10:43.127204 29276 leader_election.cc:288] T 00000000000000000000000000000000 P e527a90723fc4a0288ad21cb7d0379cc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 06:10:43.127285 29281 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e527a90723fc4a0288ad21cb7d0379cc [term 1 FOLLOWER]: Leader election won for term 1
I20260620 06:10:43.127524 29281 raft_consensus.cc:686] T 00000000000000000000000000000000 P e527a90723fc4a0288ad21cb7d0379cc [term 1 LEADER]: Becoming Leader. State: Replica: e527a90723fc4a0288ad21cb7d0379cc, State: Running, Role: LEADER
I20260620 06:10:43.127864 29281 consensus_queue.cc:237] T 00000000000000000000000000000000 P e527a90723fc4a0288ad21cb7d0379cc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e527a90723fc4a0288ad21cb7d0379cc" member_type: VOTER last_known_addr { host: "127.11.102.62" port: 32993 } }
I20260620 06:10:43.128399 29276 sys_catalog.cc:564] T 00000000000000000000000000000000 P e527a90723fc4a0288ad21cb7d0379cc [sys.catalog]: configured and running, proceeding with master startup.
I20260620 06:10:43.129448 29282 sys_catalog.cc:455] T 00000000000000000000000000000000 P e527a90723fc4a0288ad21cb7d0379cc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e527a90723fc4a0288ad21cb7d0379cc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e527a90723fc4a0288ad21cb7d0379cc" member_type: VOTER last_known_addr { host: "127.11.102.62" port: 32993 } } }
I20260620 06:10:43.129572 29282 sys_catalog.cc:458] T 00000000000000000000000000000000 P e527a90723fc4a0288ad21cb7d0379cc [sys.catalog]: This master's current role is: LEADER
I20260620 06:10:43.129837 29283 sys_catalog.cc:455] T 00000000000000000000000000000000 P e527a90723fc4a0288ad21cb7d0379cc [sys.catalog]: SysCatalogTable state changed. Reason: New leader e527a90723fc4a0288ad21cb7d0379cc. Latest consensus state: current_term: 1 leader_uuid: "e527a90723fc4a0288ad21cb7d0379cc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e527a90723fc4a0288ad21cb7d0379cc" member_type: VOTER last_known_addr { host: "127.11.102.62" port: 32993 } } }
I20260620 06:10:43.129925 29283 sys_catalog.cc:458] T 00000000000000000000000000000000 P e527a90723fc4a0288ad21cb7d0379cc [sys.catalog]: This master's current role is: LEADER
I20260620 06:10:43.129947 29290 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 06:10:43.133104 29290 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 06:10:43.138665 29290 catalog_manager.cc:1347] Generated new cluster ID: 74bdbb76e0ee40d3a14ca645f91b0d86
I20260620 06:10:43.138739 29290 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 06:10:43.155151 29290 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 06:10:43.156038 29290 catalog_manager.cc:1504] Loading token signing keys...
I20260620 06:10:43.164163 29290 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e527a90723fc4a0288ad21cb7d0379cc: Generated new TSK 0
I20260620 06:10:43.164920 29290 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 06:10:43.171938 11672 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9HMp43/build/debug/bin/kudu
/tmp/dist-test-task9HMp43/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781935603120653-11672-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781935603120653-11672-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781935603120653-11672-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781935603120653-11672-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.102.1:0
--local_ip_for_outbound_sockets=127.11.102.1
--webserver_interface=127.11.102.1
--webserver_port=0
--tserver_master_addrs=127.11.102.62:32993
--builtin_ntp_servers=127.11.102.20:33837
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260620 06:10:43.292070 29300 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:10:43.292339 29300 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:10:43.292387 29300 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:10:43.296482 29300 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:10:43.296541 29300 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:10:43.296597 29300 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.102.1
I20260620 06:10:43.301474 29300 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.102.20:33837
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781935603120653-11672-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781935603120653-11672-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.102.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781935603120653-11672-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.11.102.1
--webserver_port=0
--tserver_master_addrs=127.11.102.62:32993
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29300
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.102.1
--log_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781935603120653-11672-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 05:28:48 UTC on ccycloud.quasar-fixhms.root.comops.site
build id 4457020
I20260620 06:10:43.302574 29300 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:10:43.303958 29300 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 06:10:43.313095 29308 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:10:43.313107 29306 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:10:43.313107 29305 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:10:43.313175 29300 server_base.cc:1034] running on GCE node
I20260620 06:10:43.313737 29300 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:10:43.314330 29300 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:10:43.315529 29300 hybrid_clock.cc:648] HybridClock initialized: now 1781935843315511 us; error 28 us; skew 500 ppm
I20260620 06:10:43.317469 29300 webserver.cc:460] Webserver started at http://127.11.102.1:35267/ using document root <none> and password file <none>
I20260620 06:10:43.318112 29300 fs_manager.cc:359] Metadata directory not provided
I20260620 06:10:43.318173 29300 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:10:43.318384 29300 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 06:10:43.320662 29300 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781935603120653-11672-0/minicluster-data/ts-0/data/instance:
uuid: "0556b29b275242f79f55e2c997e91463"
format_stamp: "Formatted at 2026-06-20 06:10:43 on dist-test-slave-9hm5"
server_key: "209b1990f8ead5991c38df54d3cdad29"
server_key_iv: "3c12be6562c4ef4f5f3ea78471f8d41c"
server_key_version: "encryptionkey@0"
I20260620 06:10:43.321244 29300 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781935603120653-11672-0/minicluster-data/ts-0/wal/instance:
uuid: "0556b29b275242f79f55e2c997e91463"
format_stamp: "Formatted at 2026-06-20 06:10:43 on dist-test-slave-9hm5"
server_key: "209b1990f8ead5991c38df54d3cdad29"
server_key_iv: "3c12be6562c4ef4f5f3ea78471f8d41c"
server_key_version: "encryptionkey@0"
I20260620 06:10:43.324962 29300 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 06:10:43.327344 29314 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:10:43.328615 29300 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 06:10:43.328737 29300 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781935603120653-11672-0/minicluster-data/ts-0/data,/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781935603120653-11672-0/minicluster-data/ts-0/wal
uuid: "0556b29b275242f79f55e2c997e91463"
format_stamp: "Formatted at 2026-06-20 06:10:43 on dist-test-slave-9hm5"
server_key: "209b1990f8ead5991c38df54d3cdad29"
server_key_iv: "3c12be6562c4ef4f5f3ea78471f8d41c"
server_key_version: "encryptionkey@0"
I20260620 06:10:43.328866 29300 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781935603120653-11672-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781935603120653-11672-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781935603120653-11672-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:10:43.346254 29300 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:10:43.347008 29300 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:10:43.347194 29300 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:10:43.347788 29300 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 06:10:43.348829 29300 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 06:10:43.348893 29300 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 06:10:43.348956 29300 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 06:10:43.349004 29300 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 06:10:43.357967 29300 rpc_server.cc:307] RPC server started. Bound to: 127.11.102.1:44647
I20260620 06:10:43.357998 29427 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.102.1:44647 every 8 connection(s)
I20260620 06:10:43.358938 29300 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781935603120653-11672-0/minicluster-data/ts-0/data/info.pb
I20260620 06:10:43.368403 29279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:10:43.360802 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.102.1:35829 (local address 127.11.102.62:32993)
0620 06:10:43.360984 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0620 06:10:43.360989 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 06:10:43.362115 (+ 1126us) server_negotiation.cc:407] Connection header received
0620 06:10:43.363531 (+ 1416us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:10:43.363534 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:10:43.363582 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:10:43.363683 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:10:43.364609 (+ 926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:10:43.365177 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:10:43.366336 (+ 1159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:10:43.366583 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:10:43.366676 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:10:43.367898 (+ 1222us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:10:43.367902 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:10:43.367904 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 06:10:43.367922 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:10:43.367944 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 06:10:43.367960 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:10:43.368030 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0620 06:10:43.368222 (+ 192us) server_negotiation.cc:299] Negotiation successful
0620 06:10:43.368280 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260620 06:10:43.368857 11672 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9HMp43/build/debug/bin/kudu as pid 29300
I20260620 06:10:43.369020 11672 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781935603120653-11672-0/minicluster-data/ts-0/wal/instance
I20260620 06:10:43.369316 11672 external_mini_cluster.cc:1428] Setting key 0ab133bad2c0ffb33612f57ef9e78703
I20260620 06:10:43.369340 29430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:10:43.361254 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.102.62:32993 (local address 127.11.102.1:35829)
0620 06:10:43.361969 (+ 715us) negotiation.cc:107] Waiting for socket to connect
0620 06:10:43.362011 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0620 06:10:43.363288 (+ 1277us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:10:43.363852 (+ 564us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:10:43.363859 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:10:43.363945 (+ 86us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:10:43.364451 (+ 506us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:10:43.364466 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:10:43.365282 (+ 816us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:10:43.365286 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:10:43.366198 (+ 912us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:10:43.366209 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:10:43.366748 (+ 539us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:10:43.366751 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:10:43.366899 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:10:43.367549 (+ 650us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 06:10:43.367550 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:10:43.367592 (+ 42us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 06:10:43.367724 (+ 132us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:10:43.368071 (+ 347us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:10:43.368078 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:10:43.368083 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 06:10:43.368209 (+ 126us) client_negotiation.cc:240] Negotiation successful
0620 06:10:43.368452 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":318,"thread_start_us":135,"threads_started":1}
I20260620 06:10:43.370682 29428 heartbeater.cc:347] Connected to a master server at 127.11.102.62:32993
I20260620 06:10:43.370935 29428 heartbeater.cc:464] Registering TS with master...
I20260620 06:10:43.371483 11672 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9HMp43/build/debug/bin/kudu
/tmp/dist-test-task9HMp43/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781935603120653-11672-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781935603120653-11672-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781935603120653-11672-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781935603120653-11672-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.102.2:0
--local_ip_for_outbound_sockets=127.11.102.2
--webserver_interface=127.11.102.2
--webserver_port=0
--tserver_master_addrs=127.11.102.62:32993
--builtin_ntp_servers=127.11.102.20:33837
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260620 06:10:43.371510 29428 heartbeater.cc:511] Master 127.11.102.62:32993 requested a full tablet report, sending...
I20260620 06:10:43.373039 29239 ts_manager.cc:194] Registered new tserver with Master: 0556b29b275242f79f55e2c997e91463 (127.11.102.1:44647)
I20260620 06:10:43.374567 29239 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.11.102.1:35829
W20260620 06:10:43.485087 29431 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:10:43.485380 29431 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:10:43.485421 29431 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:10:43.489588 29431 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:10:43.489645 29431 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:10:43.489699 29431 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.102.2
I20260620 06:10:43.494669 29431 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.102.20:33837
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781935603120653-11672-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781935603120653-11672-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.102.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781935603120653-11672-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.11.102.2
--webserver_port=0
--tserver_master_addrs=127.11.102.62:32993
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29431
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.102.2
--log_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781935603120653-11672-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 05:28:48 UTC on ccycloud.quasar-fixhms.root.comops.site
build id 4457020
I20260620 06:10:43.495798 29431 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:10:43.497149 29431 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 06:10:43.507126 29437 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:10:43.507126 29436 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:10:43.507349 29439 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:10:43.507386 29431 server_base.cc:1034] running on GCE node
I20260620 06:10:43.507975 29431 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:10:43.508611 29431 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:10:43.509822 29431 hybrid_clock.cc:648] HybridClock initialized: now 1781935843509807 us; error 34 us; skew 500 ppm
I20260620 06:10:43.511691 29431 webserver.cc:460] Webserver started at http://127.11.102.2:42401/ using document root <none> and password file <none>
I20260620 06:10:43.512260 29431 fs_manager.cc:359] Metadata directory not provided
I20260620 06:10:43.512352 29431 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:10:43.512573 29431 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 06:10:43.514550 29431 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781935603120653-11672-0/minicluster-data/ts-1/data/instance:
uuid: "557a4b2d837544e29cf86bf3bae88fa3"
format_stamp: "Formatted at 2026-06-20 06:10:43 on dist-test-slave-9hm5"
server_key: "905362bdd83ec0d9a9393aec9133f922"
server_key_iv: "cff50ec510515d2d0b2c1c50523c2f9b"
server_key_version: "encryptionkey@0"
I20260620 06:10:43.515108 29431 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781935603120653-11672-0/minicluster-data/ts-1/wal/instance:
uuid: "557a4b2d837544e29cf86bf3bae88fa3"
format_stamp: "Formatted at 2026-06-20 06:10:43 on dist-test-slave-9hm5"
server_key: "905362bdd83ec0d9a9393aec9133f922"
server_key_iv: "cff50ec510515d2d0b2c1c50523c2f9b"
server_key_version: "encryptionkey@0"
I20260620 06:10:43.518903 29431 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260620 06:10:43.521346 29445 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:10:43.522558 29431 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260620 06:10:43.522696 29431 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781935603120653-11672-0/minicluster-data/ts-1/data,/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781935603120653-11672-0/minicluster-data/ts-1/wal
uuid: "557a4b2d837544e29cf86bf3bae88fa3"
format_stamp: "Formatted at 2026-06-20 06:10:43 on dist-test-slave-9hm5"
server_key: "905362bdd83ec0d9a9393aec9133f922"
server_key_iv: "cff50ec510515d2d0b2c1c50523c2f9b"
server_key_version: "encryptionkey@0"
I20260620 06:10:43.522807 29431 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781935603120653-11672-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781935603120653-11672-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781935603120653-11672-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:10:43.542775 29431 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:10:43.543495 29431 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:10:43.543718 29431 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:10:43.544318 29431 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 06:10:43.545233 29431 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 06:10:43.545302 29431 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 06:10:43.545382 29431 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 06:10:43.545432 29431 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 06:10:43.554447 29431 rpc_server.cc:307] RPC server started. Bound to: 127.11.102.2:44517
I20260620 06:10:43.554478 29558 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.102.2:44517 every 8 connection(s)
I20260620 06:10:43.555291 29431 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781935603120653-11672-0/minicluster-data/ts-1/data/info.pb
I20260620 06:10:43.558702 11672 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9HMp43/build/debug/bin/kudu as pid 29431
I20260620 06:10:43.558802 11672 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781935603120653-11672-0/minicluster-data/ts-1/wal/instance
I20260620 06:10:43.559119 11672 external_mini_cluster.cc:1428] Setting key ba794897f214eaf3831310c6bb19d308
I20260620 06:10:43.561050 11672 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9HMp43/build/debug/bin/kudu
/tmp/dist-test-task9HMp43/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781935603120653-11672-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781935603120653-11672-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781935603120653-11672-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781935603120653-11672-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.102.3:0
--local_ip_for_outbound_sockets=127.11.102.3
--webserver_interface=127.11.102.3
--webserver_port=0
--tserver_master_addrs=127.11.102.62:32993
--builtin_ntp_servers=127.11.102.20:33837
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260620 06:10:43.564725 29279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:10:43.557092 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.102.2:35897 (local address 127.11.102.62:32993)
0620 06:10:43.557237 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0620 06:10:43.557241 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 06:10:43.558267 (+ 1026us) server_negotiation.cc:407] Connection header received
0620 06:10:43.559842 (+ 1575us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:10:43.559845 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:10:43.559900 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:10:43.560008 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:10:43.561011 (+ 1003us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:10:43.561611 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:10:43.562743 (+ 1132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:10:43.562968 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:10:43.563084 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:10:43.564229 (+ 1145us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:10:43.564232 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:10:43.564235 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 06:10:43.564248 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:10:43.564276 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 06:10:43.564285 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:10:43.564361 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0620 06:10:43.564522 (+ 161us) server_negotiation.cc:299] Negotiation successful
0620 06:10:43.564582 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":21,"server-negotiator.queue_time_us":60}
I20260620 06:10:43.565591 29561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:10:43.557457 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.102.62:32993 (local address 127.11.102.2:35897)
0620 06:10:43.558075 (+ 618us) negotiation.cc:107] Waiting for socket to connect
0620 06:10:43.558114 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0620 06:10:43.559600 (+ 1486us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:10:43.560191 (+ 591us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:10:43.560199 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:10:43.560283 (+ 84us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:10:43.560865 (+ 582us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:10:43.560879 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:10:43.561742 (+ 863us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:10:43.561746 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:10:43.562619 (+ 873us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:10:43.562628 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:10:43.563123 (+ 495us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:10:43.563127 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:10:43.563291 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:10:43.563972 (+ 681us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 06:10:43.563973 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:10:43.564012 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 06:10:43.564076 (+ 64us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:10:43.564375 (+ 299us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:10:43.564386 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:10:43.564392 (+ 6us) client_negotiation.cc:769] Sending connection context
0620 06:10:43.564521 (+ 129us) client_negotiation.cc:240] Negotiation successful
0620 06:10:43.564808 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":248,"thread_start_us":108,"threads_started":1}
I20260620 06:10:43.566742 29559 heartbeater.cc:347] Connected to a master server at 127.11.102.62:32993
I20260620 06:10:43.566998 29559 heartbeater.cc:464] Registering TS with master...
I20260620 06:10:43.567517 29559 heartbeater.cc:511] Master 127.11.102.62:32993 requested a full tablet report, sending...
I20260620 06:10:43.568608 29239 ts_manager.cc:194] Registered new tserver with Master: 557a4b2d837544e29cf86bf3bae88fa3 (127.11.102.2:44517)
I20260620 06:10:43.569485 29239 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.11.102.2:35897
W20260620 06:10:43.673641 29562 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:10:43.673898 29562 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:10:43.673935 29562 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:10:43.678063 29562 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:10:43.678113 29562 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:10:43.678167 29562 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.102.3
I20260620 06:10:43.682780 29562 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.102.20:33837
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781935603120653-11672-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781935603120653-11672-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.102.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781935603120653-11672-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.11.102.3
--webserver_port=0
--tserver_master_addrs=127.11.102.62:32993
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29562
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.102.3
--log_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781935603120653-11672-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 05:28:48 UTC on ccycloud.quasar-fixhms.root.comops.site
build id 4457020
I20260620 06:10:43.683797 29562 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:10:43.685086 29562 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 06:10:43.694173 29570 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:10:43.694271 29567 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:10:43.694237 29568 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:10:43.694650 29562 server_base.cc:1034] running on GCE node
I20260620 06:10:43.695019 29562 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:10:43.695649 29562 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:10:43.696830 29562 hybrid_clock.cc:648] HybridClock initialized: now 1781935843696814 us; error 29 us; skew 500 ppm
I20260620 06:10:43.698549 29562 webserver.cc:460] Webserver started at http://127.11.102.3:37689/ using document root <none> and password file <none>
I20260620 06:10:43.699103 29562 fs_manager.cc:359] Metadata directory not provided
I20260620 06:10:43.699163 29562 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:10:43.699405 29562 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 06:10:43.701467 29562 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781935603120653-11672-0/minicluster-data/ts-2/data/instance:
uuid: "237e1cbd765f41fc8a2d8afd045d418a"
format_stamp: "Formatted at 2026-06-20 06:10:43 on dist-test-slave-9hm5"
server_key: "04ac07c88cdca96f55115073c028f957"
server_key_iv: "051dcb876102685fea97161782b4144b"
server_key_version: "encryptionkey@0"
I20260620 06:10:43.701992 29562 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781935603120653-11672-0/minicluster-data/ts-2/wal/instance:
uuid: "237e1cbd765f41fc8a2d8afd045d418a"
format_stamp: "Formatted at 2026-06-20 06:10:43 on dist-test-slave-9hm5"
server_key: "04ac07c88cdca96f55115073c028f957"
server_key_iv: "051dcb876102685fea97161782b4144b"
server_key_version: "encryptionkey@0"
I20260620 06:10:43.705726 29562 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.004s
I20260620 06:10:43.708026 29576 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:10:43.709120 29562 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 06:10:43.709247 29562 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781935603120653-11672-0/minicluster-data/ts-2/data,/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781935603120653-11672-0/minicluster-data/ts-2/wal
uuid: "237e1cbd765f41fc8a2d8afd045d418a"
format_stamp: "Formatted at 2026-06-20 06:10:43 on dist-test-slave-9hm5"
server_key: "04ac07c88cdca96f55115073c028f957"
server_key_iv: "051dcb876102685fea97161782b4144b"
server_key_version: "encryptionkey@0"
I20260620 06:10:43.709350 29562 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781935603120653-11672-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781935603120653-11672-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781935603120653-11672-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:10:43.738903 29562 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:10:43.739653 29562 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:10:43.739861 29562 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:10:43.740432 29562 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 06:10:43.741354 29562 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 06:10:43.741423 29562 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 06:10:43.741518 29562 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 06:10:43.741569 29562 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 06:10:43.749986 29562 rpc_server.cc:307] RPC server started. Bound to: 127.11.102.3:34167
I20260620 06:10:43.750015 29689 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.102.3:34167 every 8 connection(s)
I20260620 06:10:43.750939 29562 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781935603120653-11672-0/minicluster-data/ts-2/data/info.pb
I20260620 06:10:43.757622 11672 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9HMp43/build/debug/bin/kudu as pid 29562
I20260620 06:10:43.757709 11672 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781935603120653-11672-0/minicluster-data/ts-2/wal/instance
I20260620 06:10:43.757999 11672 external_mini_cluster.cc:1428] Setting key 2e862de2a6f683457f3b7a59ea02d37d
I20260620 06:10:43.760013 29279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:10:43.752738 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.102.3:44609 (local address 127.11.102.62:32993)
0620 06:10:43.752874 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0620 06:10:43.752877 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 06:10:43.753889 (+ 1012us) server_negotiation.cc:407] Connection header received
0620 06:10:43.755262 (+ 1373us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:10:43.755266 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:10:43.755323 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:10:43.755428 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:10:43.756301 (+ 873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:10:43.756819 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:10:43.758014 (+ 1195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:10:43.758300 (+ 286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:10:43.758415 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:10:43.759499 (+ 1084us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:10:43.759503 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:10:43.759506 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 06:10:43.759526 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:10:43.759544 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 06:10:43.759552 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:10:43.759649 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0620 06:10:43.759810 (+ 161us) server_negotiation.cc:299] Negotiation successful
0620 06:10:43.759869 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260620 06:10:43.760784 29692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:10:43.753130 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.102.62:32993 (local address 127.11.102.3:44609)
0620 06:10:43.753741 (+ 611us) negotiation.cc:107] Waiting for socket to connect
0620 06:10:43.753780 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0620 06:10:43.755058 (+ 1278us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:10:43.755598 (+ 540us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:10:43.755605 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:10:43.755705 (+ 100us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:10:43.756157 (+ 452us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:10:43.756169 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:10:43.756955 (+ 786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:10:43.756959 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:10:43.757895 (+ 936us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:10:43.757905 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:10:43.758458 (+ 553us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:10:43.758462 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:10:43.758619 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:10:43.759241 (+ 622us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 06:10:43.759242 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:10:43.759281 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 06:10:43.759338 (+ 57us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:10:43.759651 (+ 313us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:10:43.759663 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:10:43.759672 (+ 9us) client_negotiation.cc:769] Sending connection context
0620 06:10:43.759807 (+ 135us) client_negotiation.cc:240] Negotiation successful
0620 06:10:43.760069 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":249,"thread_start_us":105,"threads_started":1}
I20260620 06:10:43.761773 29690 heartbeater.cc:347] Connected to a master server at 127.11.102.62:32993
I20260620 06:10:43.762015 29690 heartbeater.cc:464] Registering TS with master...
I20260620 06:10:43.762540 29690 heartbeater.cc:511] Master 127.11.102.62:32993 requested a full tablet report, sending...
I20260620 06:10:43.763604 29239 ts_manager.cc:194] Registered new tserver with Master: 237e1cbd765f41fc8a2d8afd045d418a (127.11.102.3:34167)
I20260620 06:10:43.764194 29239 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.11.102.3:44609
I20260620 06:10:43.771718 11672 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 06:10:43.777020 29279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:10:43.773589 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36108 (local address 127.11.102.62:32993)
0620 06:10:43.773700 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0620 06:10:43.773703 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 06:10:43.773879 (+ 176us) server_negotiation.cc:407] Connection header received
0620 06:10:43.773968 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:10:43.773970 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:10:43.774007 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:10:43.774069 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:10:43.774279 (+ 210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:10:43.774764 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:10:43.775541 (+ 777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:10:43.775792 (+ 251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:10:43.775868 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:10:43.776613 (+ 745us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:10:43.776617 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:10:43.776619 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 06:10:43.776630 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:10:43.776651 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 06:10:43.776659 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:10:43.776752 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0620 06:10:43.776869 (+ 117us) server_negotiation.cc:299] Negotiation successful
0620 06:10:43.776918 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260620 06:10:43.779719 29239 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36108:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 06:10:43.782717 29239 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 06:10:43.796782 29704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:10:43.791658 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33946 (local address 127.11.102.2:44517)
0620 06:10:43.792023 (+ 365us) server_negotiation.cc:206] Beginning negotiation
0620 06:10:43.792036 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 06:10:43.792060 (+ 24us) server_negotiation.cc:407] Connection header received
0620 06:10:43.792186 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:10:43.792192 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:10:43.792410 (+ 218us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:10:43.792555 (+ 145us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 06:10:43.793046 (+ 491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:10:43.793942 (+ 896us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:10:43.795525 (+ 1583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:10:43.796150 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:10:43.796275 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:10:43.796482 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0620 06:10:43.796546 (+ 64us) server_negotiation.cc:299] Negotiation successful
0620 06:10:43.796635 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":254,"thread_start_us":147,"threads_started":1}
I20260620 06:10:43.796778 29702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:10:43.791115 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.102.2:44517 (local address 127.0.0.1:33946)
0620 06:10:43.791975 (+ 860us) negotiation.cc:107] Waiting for socket to connect
0620 06:10:43.791988 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 06:10:43.792070 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:10:43.792688 (+ 618us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:10:43.792694 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:10:43.792715 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 06:10:43.792914 (+ 199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:10:43.792921 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:10:43.794055 (+ 1134us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:10:43.794061 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:10:43.795386 (+ 1325us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:10:43.795396 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:10:43.796276 (+ 880us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:10:43.796279 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:10:43.796381 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:10:43.796416 (+ 35us) client_negotiation.cc:769] Sending connection context
0620 06:10:43.796512 (+ 96us) client_negotiation.cc:240] Negotiation successful
0620 06:10:43.796625 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":798,"mutex_wait_us":23,"thread_start_us":101,"threads_started":1}
I20260620 06:10:43.796984 29701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:10:43.791039 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.102.3:34167 (local address 127.0.0.1:56462)
0620 06:10:43.791479 (+ 440us) negotiation.cc:107] Waiting for socket to connect
0620 06:10:43.791525 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0620 06:10:43.791675 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:10:43.792762 (+ 1087us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:10:43.792765 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:10:43.792782 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 06:10:43.792935 (+ 153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:10:43.792942 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:10:43.794380 (+ 1438us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:10:43.794383 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:10:43.795476 (+ 1093us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:10:43.795486 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:10:43.796586 (+ 1100us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:10:43.796590 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:10:43.796689 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:10:43.796712 (+ 23us) client_negotiation.cc:769] Sending connection context
0620 06:10:43.796764 (+ 52us) client_negotiation.cc:240] Negotiation successful
0620 06:10:43.796831 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":275,"mutex_wait_us":24,"thread_start_us":81,"threads_started":1}
I20260620 06:10:43.797176 29705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:10:43.791737 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.102.1:44647 (local address 127.0.0.1:53118)
0620 06:10:43.792580 (+ 843us) negotiation.cc:107] Waiting for socket to connect
0620 06:10:43.792594 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 06:10:43.792689 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:10:43.793199 (+ 510us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:10:43.793202 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:10:43.793220 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 06:10:43.793292 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:10:43.793298 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:10:43.794595 (+ 1297us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:10:43.794598 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:10:43.795599 (+ 1001us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:10:43.795609 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:10:43.796797 (+ 1188us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:10:43.796800 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:10:43.796888 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:10:43.796908 (+ 20us) client_negotiation.cc:769] Sending connection context
0620 06:10:43.796973 (+ 65us) client_negotiation.cc:240] Negotiation successful
0620 06:10:43.797045 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":786,"mutex_wait_us":1,"thread_start_us":90,"threads_started":1}
I20260620 06:10:43.797609 29706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:10:43.792188 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53118 (local address 127.11.102.1:44647)
0620 06:10:43.792597 (+ 409us) server_negotiation.cc:206] Beginning negotiation
0620 06:10:43.792609 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 06:10:43.792777 (+ 168us) server_negotiation.cc:407] Connection header received
0620 06:10:43.792856 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:10:43.792860 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:10:43.793066 (+ 206us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:10:43.793213 (+ 147us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 06:10:43.793402 (+ 189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:10:43.794469 (+ 1067us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:10:43.795743 (+ 1274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:10:43.796463 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:10:43.796565 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:10:43.796740 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0620 06:10:43.797342 (+ 602us) server_negotiation.cc:299] Negotiation successful
0620 06:10:43.797455 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":313,"thread_start_us":140,"threads_started":1}
I20260620 06:10:43.797971 29703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:10:43.791145 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56462 (local address 127.11.102.3:34167)
0620 06:10:43.792311 (+ 1166us) server_negotiation.cc:206] Beginning negotiation
0620 06:10:43.792324 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 06:10:43.792345 (+ 21us) server_negotiation.cc:407] Connection header received
0620 06:10:43.792411 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:10:43.792417 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:10:43.792587 (+ 170us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:10:43.792765 (+ 178us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 06:10:43.793076 (+ 311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:10:43.794249 (+ 1173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:10:43.795573 (+ 1324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:10:43.796442 (+ 869us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:10:43.797466 (+ 1024us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:10:43.797650 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0620 06:10:43.797713 (+ 63us) server_negotiation.cc:299] Negotiation successful
0620 06:10:43.797802 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1072,"thread_start_us":86,"threads_started":1}
I20260620 06:10:43.799896 29493 tablet_service.cc:1467] Processing CreateTablet for tablet 37c3be17c54147ecb92acfad442037de (DEFAULT_TABLE table=test-table [id=1eeeb20383464ec7b60828c72f607bca]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 06:10:43.800848 29493 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 37c3be17c54147ecb92acfad442037de. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 06:10:43.800505 29362 tablet_service.cc:1467] Processing CreateTablet for tablet 37c3be17c54147ecb92acfad442037de (DEFAULT_TABLE table=test-table [id=1eeeb20383464ec7b60828c72f607bca]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 06:10:43.800505 29624 tablet_service.cc:1467] Processing CreateTablet for tablet 37c3be17c54147ecb92acfad442037de (DEFAULT_TABLE table=test-table [id=1eeeb20383464ec7b60828c72f607bca]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 06:10:43.801507 29362 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 37c3be17c54147ecb92acfad442037de. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 06:10:43.801507 29624 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 37c3be17c54147ecb92acfad442037de. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 06:10:43.806648 29707 tablet_bootstrap.cc:492] T 37c3be17c54147ecb92acfad442037de P 557a4b2d837544e29cf86bf3bae88fa3: Bootstrap starting.
I20260620 06:10:43.808465 29709 tablet_bootstrap.cc:492] T 37c3be17c54147ecb92acfad442037de P 0556b29b275242f79f55e2c997e91463: Bootstrap starting.
I20260620 06:10:43.808846 29708 tablet_bootstrap.cc:492] T 37c3be17c54147ecb92acfad442037de P 237e1cbd765f41fc8a2d8afd045d418a: Bootstrap starting.
I20260620 06:10:43.809653 29707 tablet_bootstrap.cc:654] T 37c3be17c54147ecb92acfad442037de P 557a4b2d837544e29cf86bf3bae88fa3: Neither blocks nor log segments found. Creating new log.
I20260620 06:10:43.810503 29707 log.cc:824] T 37c3be17c54147ecb92acfad442037de P 557a4b2d837544e29cf86bf3bae88fa3: Log is configured to *not* fsync() on all Append() calls
I20260620 06:10:43.810943 29708 tablet_bootstrap.cc:654] T 37c3be17c54147ecb92acfad442037de P 237e1cbd765f41fc8a2d8afd045d418a: Neither blocks nor log segments found. Creating new log.
I20260620 06:10:43.811790 29708 log.cc:824] T 37c3be17c54147ecb92acfad442037de P 237e1cbd765f41fc8a2d8afd045d418a: Log is configured to *not* fsync() on all Append() calls
I20260620 06:10:43.812498 29707 tablet_bootstrap.cc:492] T 37c3be17c54147ecb92acfad442037de P 557a4b2d837544e29cf86bf3bae88fa3: No bootstrap required, opened a new log
I20260620 06:10:43.812767 29707 ts_tablet_manager.cc:1397] T 37c3be17c54147ecb92acfad442037de P 557a4b2d837544e29cf86bf3bae88fa3: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260620 06:10:43.812985 29709 tablet_bootstrap.cc:654] T 37c3be17c54147ecb92acfad442037de P 0556b29b275242f79f55e2c997e91463: Neither blocks nor log segments found. Creating new log.
I20260620 06:10:43.813771 29709 log.cc:824] T 37c3be17c54147ecb92acfad442037de P 0556b29b275242f79f55e2c997e91463: Log is configured to *not* fsync() on all Append() calls
I20260620 06:10:43.813814 29708 tablet_bootstrap.cc:492] T 37c3be17c54147ecb92acfad442037de P 237e1cbd765f41fc8a2d8afd045d418a: No bootstrap required, opened a new log
I20260620 06:10:43.813977 29708 ts_tablet_manager.cc:1397] T 37c3be17c54147ecb92acfad442037de P 237e1cbd765f41fc8a2d8afd045d418a: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.004s
I20260620 06:10:43.815587 29709 tablet_bootstrap.cc:492] T 37c3be17c54147ecb92acfad442037de P 0556b29b275242f79f55e2c997e91463: No bootstrap required, opened a new log
I20260620 06:10:43.815793 29709 ts_tablet_manager.cc:1397] T 37c3be17c54147ecb92acfad442037de P 0556b29b275242f79f55e2c997e91463: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260620 06:10:43.816290 29707 raft_consensus.cc:348] T 37c3be17c54147ecb92acfad442037de P 557a4b2d837544e29cf86bf3bae88fa3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "237e1cbd765f41fc8a2d8afd045d418a" member_type: VOTER last_known_addr { host: "127.11.102.3" port: 34167 } } peers { permanent_uuid: "557a4b2d837544e29cf86bf3bae88fa3" member_type: VOTER last_known_addr { host: "127.11.102.2" port: 44517 } } peers { permanent_uuid: "0556b29b275242f79f55e2c997e91463" member_type: VOTER last_known_addr { host: "127.11.102.1" port: 44647 } }
I20260620 06:10:43.816521 29707 raft_consensus.cc:374] T 37c3be17c54147ecb92acfad442037de P 557a4b2d837544e29cf86bf3bae88fa3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 06:10:43.816591 29707 raft_consensus.cc:729] T 37c3be17c54147ecb92acfad442037de P 557a4b2d837544e29cf86bf3bae88fa3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 557a4b2d837544e29cf86bf3bae88fa3, State: Initialized, Role: FOLLOWER
I20260620 06:10:43.817044 29707 consensus_queue.cc:260] T 37c3be17c54147ecb92acfad442037de P 557a4b2d837544e29cf86bf3bae88fa3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "237e1cbd765f41fc8a2d8afd045d418a" member_type: VOTER last_known_addr { host: "127.11.102.3" port: 34167 } } peers { permanent_uuid: "557a4b2d837544e29cf86bf3bae88fa3" member_type: VOTER last_known_addr { host: "127.11.102.2" port: 44517 } } peers { permanent_uuid: "0556b29b275242f79f55e2c997e91463" member_type: VOTER last_known_addr { host: "127.11.102.1" port: 44647 } }
I20260620 06:10:43.817665 29559 heartbeater.cc:503] Master 127.11.102.62:32993 was elected leader, sending a full tablet report...
I20260620 06:10:43.817967 29707 ts_tablet_manager.cc:1428] T 37c3be17c54147ecb92acfad442037de P 557a4b2d837544e29cf86bf3bae88fa3: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260620 06:10:43.818048 29708 raft_consensus.cc:348] T 37c3be17c54147ecb92acfad442037de P 237e1cbd765f41fc8a2d8afd045d418a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "237e1cbd765f41fc8a2d8afd045d418a" member_type: VOTER last_known_addr { host: "127.11.102.3" port: 34167 } } peers { permanent_uuid: "557a4b2d837544e29cf86bf3bae88fa3" member_type: VOTER last_known_addr { host: "127.11.102.2" port: 44517 } } peers { permanent_uuid: "0556b29b275242f79f55e2c997e91463" member_type: VOTER last_known_addr { host: "127.11.102.1" port: 44647 } }
I20260620 06:10:43.818282 29708 raft_consensus.cc:374] T 37c3be17c54147ecb92acfad442037de P 237e1cbd765f41fc8a2d8afd045d418a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 06:10:43.818337 29708 raft_consensus.cc:729] T 37c3be17c54147ecb92acfad442037de P 237e1cbd765f41fc8a2d8afd045d418a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 237e1cbd765f41fc8a2d8afd045d418a, State: Initialized, Role: FOLLOWER
I20260620 06:10:43.818750 29708 consensus_queue.cc:260] T 37c3be17c54147ecb92acfad442037de P 237e1cbd765f41fc8a2d8afd045d418a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "237e1cbd765f41fc8a2d8afd045d418a" member_type: VOTER last_known_addr { host: "127.11.102.3" port: 34167 } } peers { permanent_uuid: "557a4b2d837544e29cf86bf3bae88fa3" member_type: VOTER last_known_addr { host: "127.11.102.2" port: 44517 } } peers { permanent_uuid: "0556b29b275242f79f55e2c997e91463" member_type: VOTER last_known_addr { host: "127.11.102.1" port: 44647 } }
I20260620 06:10:43.819386 29690 heartbeater.cc:503] Master 127.11.102.62:32993 was elected leader, sending a full tablet report...
I20260620 06:10:43.819766 29708 ts_tablet_manager.cc:1428] T 37c3be17c54147ecb92acfad442037de P 237e1cbd765f41fc8a2d8afd045d418a: Time spent starting tablet: real 0.006s user 0.006s sys 0.001s
I20260620 06:10:43.821835 29709 raft_consensus.cc:348] T 37c3be17c54147ecb92acfad442037de P 0556b29b275242f79f55e2c997e91463 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "237e1cbd765f41fc8a2d8afd045d418a" member_type: VOTER last_known_addr { host: "127.11.102.3" port: 34167 } } peers { permanent_uuid: "557a4b2d837544e29cf86bf3bae88fa3" member_type: VOTER last_known_addr { host: "127.11.102.2" port: 44517 } } peers { permanent_uuid: "0556b29b275242f79f55e2c997e91463" member_type: VOTER last_known_addr { host: "127.11.102.1" port: 44647 } }
I20260620 06:10:43.822131 29709 raft_consensus.cc:374] T 37c3be17c54147ecb92acfad442037de P 0556b29b275242f79f55e2c997e91463 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 06:10:43.822191 29709 raft_consensus.cc:729] T 37c3be17c54147ecb92acfad442037de P 0556b29b275242f79f55e2c997e91463 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0556b29b275242f79f55e2c997e91463, State: Initialized, Role: FOLLOWER
I20260620 06:10:43.822670 29709 consensus_queue.cc:260] T 37c3be17c54147ecb92acfad442037de P 0556b29b275242f79f55e2c997e91463 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "237e1cbd765f41fc8a2d8afd045d418a" member_type: VOTER last_known_addr { host: "127.11.102.3" port: 34167 } } peers { permanent_uuid: "557a4b2d837544e29cf86bf3bae88fa3" member_type: VOTER last_known_addr { host: "127.11.102.2" port: 44517 } } peers { permanent_uuid: "0556b29b275242f79f55e2c997e91463" member_type: VOTER last_known_addr { host: "127.11.102.1" port: 44647 } }
I20260620 06:10:43.823330 29428 heartbeater.cc:503] Master 127.11.102.62:32993 was elected leader, sending a full tablet report...
I20260620 06:10:43.823539 29709 ts_tablet_manager.cc:1428] T 37c3be17c54147ecb92acfad442037de P 0556b29b275242f79f55e2c997e91463: Time spent starting tablet: real 0.008s user 0.005s sys 0.000s
I20260620 06:10:43.824218 29713 raft_consensus.cc:482] T 37c3be17c54147ecb92acfad442037de P 557a4b2d837544e29cf86bf3bae88fa3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 06:10:43.824395 29713 raft_consensus.cc:504] T 37c3be17c54147ecb92acfad442037de P 557a4b2d837544e29cf86bf3bae88fa3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "237e1cbd765f41fc8a2d8afd045d418a" member_type: VOTER last_known_addr { host: "127.11.102.3" port: 34167 } } peers { permanent_uuid: "557a4b2d837544e29cf86bf3bae88fa3" member_type: VOTER last_known_addr { host: "127.11.102.2" port: 44517 } } peers { permanent_uuid: "0556b29b275242f79f55e2c997e91463" member_type: VOTER last_known_addr { host: "127.11.102.1" port: 44647 } }
I20260620 06:10:43.825376 29713 leader_election.cc:288] T 37c3be17c54147ecb92acfad442037de P 557a4b2d837544e29cf86bf3bae88fa3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 237e1cbd765f41fc8a2d8afd045d418a (127.11.102.3:34167), 0556b29b275242f79f55e2c997e91463 (127.11.102.1:44647)
I20260620 06:10:43.829445 29703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:10:43.825786 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.102.2:43829 (local address 127.11.102.3:34167)
0620 06:10:43.825948 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0620 06:10:43.825951 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 06:10:43.825961 (+ 10us) server_negotiation.cc:407] Connection header received
0620 06:10:43.826031 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:10:43.826034 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:10:43.826080 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:10:43.826158 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 06:10:43.826414 (+ 256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:10:43.827167 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:10:43.828453 (+ 1286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:10:43.828932 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:10:43.829059 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:10:43.829105 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0620 06:10:43.829261 (+ 156us) server_negotiation.cc:299] Negotiation successful
0620 06:10:43.829326 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260620 06:10:43.829445 29561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:10:43.825541 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.102.3:34167 (local address 127.11.102.2:43829)
0620 06:10:43.825671 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0620 06:10:43.825689 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 06:10:43.825773 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:10:43.826166 (+ 393us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:10:43.826169 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:10:43.826184 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 06:10:43.826314 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:10:43.826322 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:10:43.827275 (+ 953us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:10:43.827281 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:10:43.828330 (+ 1049us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:10:43.828339 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:10:43.829049 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:10:43.829052 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:10:43.829148 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:10:43.829175 (+ 27us) client_negotiation.cc:769] Sending connection context
0620 06:10:43.829248 (+ 73us) client_negotiation.cc:240] Negotiation successful
0620 06:10:43.829314 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":29}
I20260620 06:10:43.829881 29706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:10:43.826224 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.102.2:60921 (local address 127.11.102.1:44647)
0620 06:10:43.826336 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0620 06:10:43.826341 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 06:10:43.826542 (+ 201us) server_negotiation.cc:407] Connection header received
0620 06:10:43.826734 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:10:43.826737 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:10:43.826780 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:10:43.826864 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 06:10:43.827226 (+ 362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:10:43.827805 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:10:43.828956 (+ 1151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:10:43.829353 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:10:43.829488 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:10:43.829530 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0620 06:10:43.829720 (+ 190us) server_negotiation.cc:299] Negotiation successful
0620 06:10:43.829784 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260620 06:10:43.829892 29716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:10:43.825827 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.102.1:44647 (local address 127.11.102.2:60921)
0620 06:10:43.826468 (+ 641us) negotiation.cc:107] Waiting for socket to connect
0620 06:10:43.826482 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 06:10:43.826604 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:10:43.826950 (+ 346us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:10:43.826953 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:10:43.826971 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 06:10:43.827092 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:10:43.827098 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:10:43.827938 (+ 840us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:10:43.827948 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:10:43.828737 (+ 789us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:10:43.828745 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:10:43.829494 (+ 749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:10:43.829498 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:10:43.829600 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:10:43.829632 (+ 32us) client_negotiation.cc:769] Sending connection context
0620 06:10:43.829696 (+ 64us) client_negotiation.cc:240] Negotiation successful
0620 06:10:43.829759 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":241,"thread_start_us":127,"threads_started":1}
I20260620 06:10:43.830127 29644 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "37c3be17c54147ecb92acfad442037de" candidate_uuid: "557a4b2d837544e29cf86bf3bae88fa3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "237e1cbd765f41fc8a2d8afd045d418a" is_pre_election: true
I20260620 06:10:43.830418 29644 raft_consensus.cc:2454] T 37c3be17c54147ecb92acfad442037de P 237e1cbd765f41fc8a2d8afd045d418a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 557a4b2d837544e29cf86bf3bae88fa3 in term 0.
I20260620 06:10:43.830478 29382 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "37c3be17c54147ecb92acfad442037de" candidate_uuid: "557a4b2d837544e29cf86bf3bae88fa3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0556b29b275242f79f55e2c997e91463" is_pre_election: true
I20260620 06:10:43.830742 29382 raft_consensus.cc:2454] T 37c3be17c54147ecb92acfad442037de P 0556b29b275242f79f55e2c997e91463 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 557a4b2d837544e29cf86bf3bae88fa3 in term 0.
I20260620 06:10:43.830840 29446 leader_election.cc:302] T 37c3be17c54147ecb92acfad442037de P 557a4b2d837544e29cf86bf3bae88fa3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 237e1cbd765f41fc8a2d8afd045d418a, 557a4b2d837544e29cf86bf3bae88fa3; no voters:
I20260620 06:10:43.831110 29713 raft_consensus.cc:2781] T 37c3be17c54147ecb92acfad442037de P 557a4b2d837544e29cf86bf3bae88fa3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 06:10:43.831204 29713 raft_consensus.cc:482] T 37c3be17c54147ecb92acfad442037de P 557a4b2d837544e29cf86bf3bae88fa3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 06:10:43.831262 29713 raft_consensus.cc:3037] T 37c3be17c54147ecb92acfad442037de P 557a4b2d837544e29cf86bf3bae88fa3 [term 0 FOLLOWER]: Advancing to term 1
I20260620 06:10:43.832317 29713 raft_consensus.cc:504] T 37c3be17c54147ecb92acfad442037de P 557a4b2d837544e29cf86bf3bae88fa3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "237e1cbd765f41fc8a2d8afd045d418a" member_type: VOTER last_known_addr { host: "127.11.102.3" port: 34167 } } peers { permanent_uuid: "557a4b2d837544e29cf86bf3bae88fa3" member_type: VOTER last_known_addr { host: "127.11.102.2" port: 44517 } } peers { permanent_uuid: "0556b29b275242f79f55e2c997e91463" member_type: VOTER last_known_addr { host: "127.11.102.1" port: 44647 } }
I20260620 06:10:43.832676 29713 leader_election.cc:288] T 37c3be17c54147ecb92acfad442037de P 557a4b2d837544e29cf86bf3bae88fa3 [CANDIDATE]: Term 1 election: Requested vote from peers 237e1cbd765f41fc8a2d8afd045d418a (127.11.102.3:34167), 0556b29b275242f79f55e2c997e91463 (127.11.102.1:44647)
I20260620 06:10:43.832988 29644 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "37c3be17c54147ecb92acfad442037de" candidate_uuid: "557a4b2d837544e29cf86bf3bae88fa3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "237e1cbd765f41fc8a2d8afd045d418a"
I20260620 06:10:43.833119 29644 raft_consensus.cc:3037] T 37c3be17c54147ecb92acfad442037de P 237e1cbd765f41fc8a2d8afd045d418a [term 0 FOLLOWER]: Advancing to term 1
I20260620 06:10:43.833132 29382 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "37c3be17c54147ecb92acfad442037de" candidate_uuid: "557a4b2d837544e29cf86bf3bae88fa3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0556b29b275242f79f55e2c997e91463"
I20260620 06:10:43.833285 29382 raft_consensus.cc:3037] T 37c3be17c54147ecb92acfad442037de P 0556b29b275242f79f55e2c997e91463 [term 0 FOLLOWER]: Advancing to term 1
I20260620 06:10:43.834254 29644 raft_consensus.cc:2454] T 37c3be17c54147ecb92acfad442037de P 237e1cbd765f41fc8a2d8afd045d418a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 557a4b2d837544e29cf86bf3bae88fa3 in term 1.
I20260620 06:10:43.834331 29382 raft_consensus.cc:2454] T 37c3be17c54147ecb92acfad442037de P 0556b29b275242f79f55e2c997e91463 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 557a4b2d837544e29cf86bf3bae88fa3 in term 1.
I20260620 06:10:43.834553 29446 leader_election.cc:302] T 37c3be17c54147ecb92acfad442037de P 557a4b2d837544e29cf86bf3bae88fa3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 237e1cbd765f41fc8a2d8afd045d418a, 557a4b2d837544e29cf86bf3bae88fa3; no voters:
I20260620 06:10:43.834725 29713 raft_consensus.cc:2781] T 37c3be17c54147ecb92acfad442037de P 557a4b2d837544e29cf86bf3bae88fa3 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 06:10:43.834940 29713 raft_consensus.cc:686] T 37c3be17c54147ecb92acfad442037de P 557a4b2d837544e29cf86bf3bae88fa3 [term 1 LEADER]: Becoming Leader. State: Replica: 557a4b2d837544e29cf86bf3bae88fa3, State: Running, Role: LEADER
I20260620 06:10:43.835266 29713 consensus_queue.cc:237] T 37c3be17c54147ecb92acfad442037de P 557a4b2d837544e29cf86bf3bae88fa3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "237e1cbd765f41fc8a2d8afd045d418a" member_type: VOTER last_known_addr { host: "127.11.102.3" port: 34167 } } peers { permanent_uuid: "557a4b2d837544e29cf86bf3bae88fa3" member_type: VOTER last_known_addr { host: "127.11.102.2" port: 44517 } } peers { permanent_uuid: "0556b29b275242f79f55e2c997e91463" member_type: VOTER last_known_addr { host: "127.11.102.1" port: 44647 } }
I20260620 06:10:43.838212 29239 catalog_manager.cc:5495] T 37c3be17c54147ecb92acfad442037de P 557a4b2d837544e29cf86bf3bae88fa3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 557a4b2d837544e29cf86bf3bae88fa3 (127.11.102.2). New cstate: current_term: 1 leader_uuid: "557a4b2d837544e29cf86bf3bae88fa3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "237e1cbd765f41fc8a2d8afd045d418a" member_type: VOTER last_known_addr { host: "127.11.102.3" port: 34167 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "557a4b2d837544e29cf86bf3bae88fa3" member_type: VOTER last_known_addr { host: "127.11.102.2" port: 44517 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0556b29b275242f79f55e2c997e91463" member_type: VOTER last_known_addr { host: "127.11.102.1" port: 44647 } health_report { overall_health: UNKNOWN } } }
I20260620 06:10:43.853390 29704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:10:43.850098 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33954 (local address 127.11.102.2:44517)
0620 06:10:43.850210 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0620 06:10:43.850213 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 06:10:43.850226 (+ 13us) server_negotiation.cc:407] Connection header received
0620 06:10:43.850362 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:10:43.850365 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:10:43.850434 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:10:43.850518 (+ 84us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 06:10:43.850750 (+ 232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:10:43.851301 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:10:43.852184 (+ 883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:10:43.852385 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:10:43.852475 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:10:43.852710 (+ 235us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 06:10:43.853108 (+ 398us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 06:10:43.853195 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0620 06:10:43.853246 (+ 51us) server_negotiation.cc:299] Negotiation successful
0620 06:10:43.853289 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260620 06:10:43.858960 29382 raft_consensus.cc:1261] T 37c3be17c54147ecb92acfad442037de P 0556b29b275242f79f55e2c997e91463 [term 1 FOLLOWER]: Refusing update from remote peer 557a4b2d837544e29cf86bf3bae88fa3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 06:10:43.858960 29644 raft_consensus.cc:1261] T 37c3be17c54147ecb92acfad442037de P 237e1cbd765f41fc8a2d8afd045d418a [term 1 FOLLOWER]: Refusing update from remote peer 557a4b2d837544e29cf86bf3bae88fa3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 06:10:43.859673 29713 consensus_queue.cc:1035] T 37c3be17c54147ecb92acfad442037de P 557a4b2d837544e29cf86bf3bae88fa3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "237e1cbd765f41fc8a2d8afd045d418a" member_type: VOTER last_known_addr { host: "127.11.102.3" port: 34167 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 06:10:43.859845 29717 consensus_queue.cc:1035] T 37c3be17c54147ecb92acfad442037de P 557a4b2d837544e29cf86bf3bae88fa3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0556b29b275242f79f55e2c997e91463" member_type: VOTER last_known_addr { host: "127.11.102.1" port: 44647 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260620 06:10:43.860360 29429 tablet.cc:2369] T 37c3be17c54147ecb92acfad442037de P 0556b29b275242f79f55e2c997e91463: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 06:10:43.867417 29722 mvcc.cc:205] Tried to move back new op lower bound from 7298809216439087104 to 7298809216352223232. Current Snapshot: MvccSnapshot[applied={T|T < 7298809216439087104}]
I20260620 06:10:43.876754 29238 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36108:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 06:10:43.876966 29238 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36108:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 06:10:43.879416 29238 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e527a90723fc4a0288ad21cb7d0379cc: Sending DeleteTablet for 3 replicas of tablet 37c3be17c54147ecb92acfad442037de
I20260620 06:10:43.880308 29362 tablet_service.cc:1514] Processing DeleteTablet for tablet 37c3be17c54147ecb92acfad442037de with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 06:10:43 UTC) from {username='slave'} at 127.0.0.1:53118
I20260620 06:10:43.880312 29493 tablet_service.cc:1514] Processing DeleteTablet for tablet 37c3be17c54147ecb92acfad442037de with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 06:10:43 UTC) from {username='slave'} at 127.0.0.1:33946
I20260620 06:10:43.880456 29624 tablet_service.cc:1514] Processing DeleteTablet for tablet 37c3be17c54147ecb92acfad442037de with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 06:10:43 UTC) from {username='slave'} at 127.0.0.1:56462
I20260620 06:10:43.880941 29734 tablet_replica.cc:330] T 37c3be17c54147ecb92acfad442037de P 237e1cbd765f41fc8a2d8afd045d418a: stopping tablet replica
I20260620 06:10:43.880966 29732 tablet_replica.cc:330] T 37c3be17c54147ecb92acfad442037de P 0556b29b275242f79f55e2c997e91463: stopping tablet replica
I20260620 06:10:43.881281 29732 raft_consensus.cc:2229] T 37c3be17c54147ecb92acfad442037de P 0556b29b275242f79f55e2c997e91463 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 06:10:43.881326 29734 raft_consensus.cc:2229] T 37c3be17c54147ecb92acfad442037de P 237e1cbd765f41fc8a2d8afd045d418a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 06:10:43.881371 11672 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9HMp43/build/debug/bin/kudu with pid 29300
I20260620 06:10:43.881688 29732 raft_consensus.cc:2258] T 37c3be17c54147ecb92acfad442037de P 0556b29b275242f79f55e2c997e91463 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 06:10:43.881951 29733 tablet_replica.cc:330] T 37c3be17c54147ecb92acfad442037de P 557a4b2d837544e29cf86bf3bae88fa3: stopping tablet replica
I20260620 06:10:43.882197 29733 raft_consensus.cc:2229] T 37c3be17c54147ecb92acfad442037de P 557a4b2d837544e29cf86bf3bae88fa3 [term 1 LEADER]: Raft consensus shutting down.
I20260620 06:10:43.882566 29734 raft_consensus.cc:2258] T 37c3be17c54147ecb92acfad442037de P 237e1cbd765f41fc8a2d8afd045d418a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 06:10:43.882582 29733 raft_consensus.cc:2258] T 37c3be17c54147ecb92acfad442037de P 557a4b2d837544e29cf86bf3bae88fa3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 06:10:43.884142 29733 ts_tablet_manager.cc:1905] T 37c3be17c54147ecb92acfad442037de P 557a4b2d837544e29cf86bf3bae88fa3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 06:10:43.884147 29734 ts_tablet_manager.cc:1905] T 37c3be17c54147ecb92acfad442037de P 237e1cbd765f41fc8a2d8afd045d418a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 06:10:43.886861 29733 ts_tablet_manager.cc:1918] T 37c3be17c54147ecb92acfad442037de P 557a4b2d837544e29cf86bf3bae88fa3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 06:10:43.886994 29733 log.cc:1196] T 37c3be17c54147ecb92acfad442037de P 557a4b2d837544e29cf86bf3bae88fa3: Deleting WAL directory at /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781935603120653-11672-0/minicluster-data/ts-1/wal/wals/37c3be17c54147ecb92acfad442037de
I20260620 06:10:43.887378 29733 ts_tablet_manager.cc:1939] T 37c3be17c54147ecb92acfad442037de P 557a4b2d837544e29cf86bf3bae88fa3: Deleting consensus metadata
I20260620 06:10:43.887553 29734 ts_tablet_manager.cc:1918] T 37c3be17c54147ecb92acfad442037de P 237e1cbd765f41fc8a2d8afd045d418a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 06:10:43.887691 29734 log.cc:1196] T 37c3be17c54147ecb92acfad442037de P 237e1cbd765f41fc8a2d8afd045d418a: Deleting WAL directory at /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781935603120653-11672-0/minicluster-data/ts-2/wal/wals/37c3be17c54147ecb92acfad442037de
I20260620 06:10:43.888055 29734 ts_tablet_manager.cc:1939] T 37c3be17c54147ecb92acfad442037de P 237e1cbd765f41fc8a2d8afd045d418a: Deleting consensus metadata
I20260620 06:10:43.888803 29227 catalog_manager.cc:4841] TS 557a4b2d837544e29cf86bf3bae88fa3 (127.11.102.2:44517): tablet 37c3be17c54147ecb92acfad442037de (table test-table [id=1eeeb20383464ec7b60828c72f607bca]) successfully deleted
I20260620 06:10:43.889232 29224 catalog_manager.cc:4841] TS 237e1cbd765f41fc8a2d8afd045d418a (127.11.102.3:34167): tablet 37c3be17c54147ecb92acfad442037de (table test-table [id=1eeeb20383464ec7b60828c72f607bca]) successfully deleted
W20260620 06:10:43.890440 29224 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.11.102.1:44647 (error 108)
I20260620 06:10:43.891064 11672 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9HMp43/build/debug/bin/kudu with pid 29431
I20260620 06:10:43.892066 29702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:10:43.891257 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.102.1:44647 (local address 127.0.0.1:53132)
0620 06:10:43.891873 (+ 616us) negotiation.cc:107] Waiting for socket to connect
0620 06:10:43.891962 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.102.1:44647: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":515}
W20260620 06:10:43.892966 29224 catalog_manager.cc:4584] TS 0556b29b275242f79f55e2c997e91463 (127.11.102.1:44647): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 37c3be17c54147ecb92acfad442037de: Network error: Client connection negotiation failed: client connection to 127.11.102.1:44647: connect: Connection refused (error 111)
I20260620 06:10:43.897353 11672 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9HMp43/build/debug/bin/kudu with pid 29562
I20260620 06:10:43.903028 11672 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9HMp43/build/debug/bin/kudu with pid 29209
2026-06-20T06:10:43Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (1005 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781935603120653-11672-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 06:10:43 dist-test-slave-9hm5 krb5kdc[29738](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 06:10:43 dist-test-slave-9hm5 krb5kdc[29738](info): set up 2 sockets
Jun 20 06:10:43 dist-test-slave-9hm5 krb5kdc[29738](info): commencing operation
krb5kdc: starting...
W20260620 06:10:44.943431 11672 mini_kdc.cc:121] Time spent starting KDC: real 1.017s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781935603120653-11672-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781935603120653-11672-0/krb5kdc/test-user.keytab.
Jun 20 06:10:45 dist-test-slave-9hm5 krb5kdc[29738](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935845, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T06:10:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T06:10:45Z Disabled control of system clock
WARNING: no policy specified for kudu/127.11.102.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.102.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.102.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781935603120653-11672-0/krb5kdc/kudu_127.11.102.62.keytab.
Entry for principal kudu/127.11.102.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781935603120653-11672-0/krb5kdc/kudu_127.11.102.62.keytab.
I20260620 06:10:45.062397 11672 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9HMp43/build/debug/bin/kudu
/tmp/dist-test-task9HMp43/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781935603120653-11672-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781935603120653-11672-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781935603120653-11672-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781935603120653-11672-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.102.62:42775
--webserver_interface=127.11.102.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781935603120653-11672-0/krb5kdc/kudu_127.11.102.62.keytab
--principal=kudu/127.11.102.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.102.20:40897
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.102.62:42775
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781935603120653-11672-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781935603120653-11672-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781935603120653-11672-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 06:10:45.169756 29751 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:10:45.170003 29751 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:10:45.170042 29751 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:10:45.173875 29751 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 06:10:45.173919 29751 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:10:45.173938 29751 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:10:45.173954 29751 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 06:10:45.173969 29751 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 06:10:45.178913 29751 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.102.20:40897
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781935603120653-11672-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781935603120653-11672-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.102.62:42775
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781935603120653-11672-0/krb5kdc/kudu_127.11.102.62.keytab
--principal=kudu/127.11.102.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.102.62:42775
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781935603120653-11672-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.102.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29751
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781935603120653-11672-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 05:28:48 UTC on ccycloud.quasar-fixhms.root.comops.site
build id 4457020
I20260620 06:10:45.179857 29751 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:10:45.180766 29751 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 06:10:45.188840 29759 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:10:45.188834 29757 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:10:45.189023 29751 server_base.cc:1034] running on GCE node
W20260620 06:10:45.188835 29756 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:10:45.189697 29751 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:10:45.190738 29751 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:10:45.191900 29751 hybrid_clock.cc:648] HybridClock initialized: now 1781935845191882 us; error 28 us; skew 500 ppm
Jun 20 06:10:45 dist-test-slave-9hm5 krb5kdc[29738](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935845, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.102.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 06:10:45.194651 29751 init.cc:377] Logged in from keytab as kudu/127.11.102.62@KRBTEST.COM (short username kudu)
I20260620 06:10:45.195500 29751 webserver.cc:460] Webserver started at http://127.11.102.62:38333/ using document root <none> and password file <none>
I20260620 06:10:45.196064 29751 fs_manager.cc:359] Metadata directory not provided
I20260620 06:10:45.196125 29751 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:10:45.196359 29751 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 06:10:45.198316 29751 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781935603120653-11672-0/minicluster-data/master-0/data/instance:
uuid: "958f6124650044afa74f9609ea59faad"
format_stamp: "Formatted at 2026-06-20 06:10:45 on dist-test-slave-9hm5"
server_key: "4409567b5cee2c02285c1c91f507cca0"
server_key_iv: "4cf557f767fa7c271a8b7eccd37729a7"
server_key_version: "encryptionkey@0"
I20260620 06:10:45.198868 29751 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781935603120653-11672-0/minicluster-data/master-0/wal/instance:
uuid: "958f6124650044afa74f9609ea59faad"
format_stamp: "Formatted at 2026-06-20 06:10:45 on dist-test-slave-9hm5"
server_key: "4409567b5cee2c02285c1c91f507cca0"
server_key_iv: "4cf557f767fa7c271a8b7eccd37729a7"
server_key_version: "encryptionkey@0"
I20260620 06:10:45.202376 29751 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260620 06:10:45.204905 29766 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:10:45.206054 29751 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260620 06:10:45.206275 29751 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781935603120653-11672-0/minicluster-data/master-0/data,/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781935603120653-11672-0/minicluster-data/master-0/wal
uuid: "958f6124650044afa74f9609ea59faad"
format_stamp: "Formatted at 2026-06-20 06:10:45 on dist-test-slave-9hm5"
server_key: "4409567b5cee2c02285c1c91f507cca0"
server_key_iv: "4cf557f767fa7c271a8b7eccd37729a7"
server_key_version: "encryptionkey@0"
I20260620 06:10:45.206383 29751 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781935603120653-11672-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781935603120653-11672-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781935603120653-11672-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:10:45.220754 29751 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:10:45.223676 29751 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:10:45.223861 29751 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:10:45.230518 29751 rpc_server.cc:307] RPC server started. Bound to: 127.11.102.62:42775
I20260620 06:10:45.230620 29818 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.102.62:42775 every 8 connection(s)
I20260620 06:10:45.231457 29751 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781935603120653-11672-0/minicluster-data/master-0/data/info.pb
I20260620 06:10:45.234078 29819 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 06:10:45.237709 11672 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9HMp43/build/debug/bin/kudu as pid 29751
I20260620 06:10:45.237874 11672 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781935603120653-11672-0/minicluster-data/master-0/wal/instance
I20260620 06:10:45.238160 11672 external_mini_cluster.cc:1428] Setting key 6e237c5176c40628027636bbdf2de68a
I20260620 06:10:45.239914 29819 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 958f6124650044afa74f9609ea59faad: Bootstrap starting.
I20260620 06:10:45.242568 29819 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 958f6124650044afa74f9609ea59faad: Neither blocks nor log segments found. Creating new log.
I20260620 06:10:45.243433 29819 log.cc:824] T 00000000000000000000000000000000 P 958f6124650044afa74f9609ea59faad: Log is configured to *not* fsync() on all Append() calls
I20260620 06:10:45.245505 29819 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 958f6124650044afa74f9609ea59faad: No bootstrap required, opened a new log
Jun 20 06:10:45 dist-test-slave-9hm5 krb5kdc[29738](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935845, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.102.62@KRBTEST.COM
I20260620 06:10:45.249931 29819 raft_consensus.cc:348] T 00000000000000000000000000000000 P 958f6124650044afa74f9609ea59faad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "958f6124650044afa74f9609ea59faad" member_type: VOTER last_known_addr { host: "127.11.102.62" port: 42775 } }
I20260620 06:10:45.250154 29819 raft_consensus.cc:374] T 00000000000000000000000000000000 P 958f6124650044afa74f9609ea59faad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 06:10:45.250187 29819 raft_consensus.cc:729] T 00000000000000000000000000000000 P 958f6124650044afa74f9609ea59faad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 958f6124650044afa74f9609ea59faad, State: Initialized, Role: FOLLOWER
I20260620 06:10:45.250744 29819 consensus_queue.cc:260] T 00000000000000000000000000000000 P 958f6124650044afa74f9609ea59faad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "958f6124650044afa74f9609ea59faad" member_type: VOTER last_known_addr { host: "127.11.102.62" port: 42775 } }
I20260620 06:10:45.250943 29819 raft_consensus.cc:388] T 00000000000000000000000000000000 P 958f6124650044afa74f9609ea59faad [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 06:10:45.251019 29819 raft_consensus.cc:482] T 00000000000000000000000000000000 P 958f6124650044afa74f9609ea59faad [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 06:10:45.251111 29819 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 958f6124650044afa74f9609ea59faad [term 0 FOLLOWER]: Advancing to term 1
I20260620 06:10:45.252211 29819 raft_consensus.cc:504] T 00000000000000000000000000000000 P 958f6124650044afa74f9609ea59faad [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "958f6124650044afa74f9609ea59faad" member_type: VOTER last_known_addr { host: "127.11.102.62" port: 42775 } }
I20260620 06:10:45.252596 29819 leader_election.cc:302] T 00000000000000000000000000000000 P 958f6124650044afa74f9609ea59faad [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 958f6124650044afa74f9609ea59faad; no voters:
I20260620 06:10:45.252902 29819 leader_election.cc:288] T 00000000000000000000000000000000 P 958f6124650044afa74f9609ea59faad [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 06:10:45.253139 29824 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 958f6124650044afa74f9609ea59faad [term 1 FOLLOWER]: Leader election won for term 1
I20260620 06:10:45.253392 29824 raft_consensus.cc:686] T 00000000000000000000000000000000 P 958f6124650044afa74f9609ea59faad [term 1 LEADER]: Becoming Leader. State: Replica: 958f6124650044afa74f9609ea59faad, State: Running, Role: LEADER
I20260620 06:10:45.253688 29824 consensus_queue.cc:237] T 00000000000000000000000000000000 P 958f6124650044afa74f9609ea59faad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "958f6124650044afa74f9609ea59faad" member_type: VOTER last_known_addr { host: "127.11.102.62" port: 42775 } }
I20260620 06:10:45.253756 29822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:10:45.239586 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35010 (local address 127.11.102.62:42775)
0620 06:10:45.240085 (+ 499us) server_negotiation.cc:206] Beginning negotiation
0620 06:10:45.240105 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0620 06:10:45.240149 (+ 44us) server_negotiation.cc:407] Connection header received
0620 06:10:45.240948 (+ 799us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:10:45.240973 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:10:45.241403 (+ 430us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:10:45.241755 (+ 352us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:10:45.242440 (+ 685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:10:45.243961 (+ 1521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:10:45.244938 (+ 977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:10:45.245594 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:10:45.245750 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:10:45.248513 (+ 2763us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:10:45.248518 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:10:45.248531 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:10:45.248557 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:10:45.250376 (+ 1819us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:10:45.250870 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:10:45.250875 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:10:45.250899 (+ 24us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:10:45.251067 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:10:45.251355 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:10:45.251358 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:10:45.251359 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:10:45.251840 (+ 481us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:10:45.251994 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0620 06:10:45.252325 (+ 331us) server_negotiation.cc:299] Negotiation successful
0620 06:10:45.252697 (+ 372us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":290,"thread_start_us":112,"threads_started":1}
I20260620 06:10:45.254247 29819 sys_catalog.cc:564] T 00000000000000000000000000000000 P 958f6124650044afa74f9609ea59faad [sys.catalog]: configured and running, proceeding with master startup.
I20260620 06:10:45.255334 29825 sys_catalog.cc:455] T 00000000000000000000000000000000 P 958f6124650044afa74f9609ea59faad [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "958f6124650044afa74f9609ea59faad" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "958f6124650044afa74f9609ea59faad" member_type: VOTER last_known_addr { host: "127.11.102.62" port: 42775 } } }
I20260620 06:10:45.255543 29825 sys_catalog.cc:458] T 00000000000000000000000000000000 P 958f6124650044afa74f9609ea59faad [sys.catalog]: This master's current role is: LEADER
I20260620 06:10:45.255834 29825 sys_catalog.cc:455] T 00000000000000000000000000000000 P 958f6124650044afa74f9609ea59faad [sys.catalog]: SysCatalogTable state changed. Reason: New leader 958f6124650044afa74f9609ea59faad. Latest consensus state: current_term: 1 leader_uuid: "958f6124650044afa74f9609ea59faad" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "958f6124650044afa74f9609ea59faad" member_type: VOTER last_known_addr { host: "127.11.102.62" port: 42775 } } }
I20260620 06:10:45.255985 29825 sys_catalog.cc:458] T 00000000000000000000000000000000 P 958f6124650044afa74f9609ea59faad [sys.catalog]: This master's current role is: LEADER
W20260620 06:10:45.259436 29839 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 958f6124650044afa74f9609ea59faad: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 06:10:45.259503 29839 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 06:10:45.259585 29833 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 06:10:45.260309 29833 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 06:10:45.265893 29833 catalog_manager.cc:1347] Generated new cluster ID: 1f1c293bb2bf4b3f9c87c624169d686a
I20260620 06:10:45.265960 29833 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 06:10:45.293915 29833 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 06:10:45.294889 29833 catalog_manager.cc:1504] Loading token signing keys...
I20260620 06:10:45.298843 29833 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 958f6124650044afa74f9609ea59faad: Generated new TSK 0
I20260620 06:10:45.299548 29833 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.11.102.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.102.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.102.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781935603120653-11672-0/krb5kdc/kudu_127.11.102.1.keytab.
Entry for principal kudu/127.11.102.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781935603120653-11672-0/krb5kdc/kudu_127.11.102.1.keytab.
I20260620 06:10:45.329929 11672 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9HMp43/build/debug/bin/kudu
/tmp/dist-test-task9HMp43/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781935603120653-11672-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781935603120653-11672-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781935603120653-11672-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781935603120653-11672-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.102.1:0
--local_ip_for_outbound_sockets=127.11.102.1
--webserver_interface=127.11.102.1
--webserver_port=0
--tserver_master_addrs=127.11.102.62:42775
--keytab_file=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781935603120653-11672-0/krb5kdc/kudu_127.11.102.1.keytab
--principal=kudu/127.11.102.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.102.20:40897
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781935603120653-11672-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781935603120653-11672-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781935603120653-11672-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 06:10:45.437086 29845 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:10:45.437326 29845 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:10:45.437409 29845 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:10:45.441381 29845 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:10:45.441437 29845 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:10:45.441552 29845 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.102.1
I20260620 06:10:45.446560 29845 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.102.20:40897
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781935603120653-11672-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781935603120653-11672-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781935603120653-11672-0/krb5kdc/kudu_127.11.102.1.keytab
--principal=kudu/127.11.102.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.102.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781935603120653-11672-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.11.102.1
--webserver_port=0
--tserver_master_addrs=127.11.102.62:42775
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29845
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.102.1
--log_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781935603120653-11672-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 05:28:48 UTC on ccycloud.quasar-fixhms.root.comops.site
build id 4457020
I20260620 06:10:45.447531 29845 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:10:45.448522 29845 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 06:10:45.458225 29851 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:10:45.458238 29850 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:10:45.458349 29845 server_base.cc:1034] running on GCE node
W20260620 06:10:45.458235 29853 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:10:45.458854 29845 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:10:45.459549 29845 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:10:45.460815 29845 hybrid_clock.cc:648] HybridClock initialized: now 1781935845460799 us; error 26 us; skew 500 ppm
Jun 20 06:10:45 dist-test-slave-9hm5 krb5kdc[29738](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935845, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.102.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 06:10:45.463725 29845 init.cc:377] Logged in from keytab as kudu/127.11.102.1@KRBTEST.COM (short username kudu)
I20260620 06:10:45.464568 29845 webserver.cc:460] Webserver started at http://127.11.102.1:35493/ using document root <none> and password file <none>
I20260620 06:10:45.465188 29845 fs_manager.cc:359] Metadata directory not provided
I20260620 06:10:45.465283 29845 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:10:45.465520 29845 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 06:10:45.467518 29845 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781935603120653-11672-0/minicluster-data/ts-0/data/instance:
uuid: "85dbfcfad0d4460395b3a0ac5c2ddc19"
format_stamp: "Formatted at 2026-06-20 06:10:45 on dist-test-slave-9hm5"
server_key: "353eaed6c5ab4c30ab33fa53992a8531"
server_key_iv: "3f78aac51504a78957696ef96f2bb368"
server_key_version: "encryptionkey@0"
I20260620 06:10:45.468130 29845 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781935603120653-11672-0/minicluster-data/ts-0/wal/instance:
uuid: "85dbfcfad0d4460395b3a0ac5c2ddc19"
format_stamp: "Formatted at 2026-06-20 06:10:45 on dist-test-slave-9hm5"
server_key: "353eaed6c5ab4c30ab33fa53992a8531"
server_key_iv: "3f78aac51504a78957696ef96f2bb368"
server_key_version: "encryptionkey@0"
I20260620 06:10:45.471714 29845 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 06:10:45.473963 29860 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:10:45.475164 29845 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 06:10:45.475293 29845 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781935603120653-11672-0/minicluster-data/ts-0/data,/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781935603120653-11672-0/minicluster-data/ts-0/wal
uuid: "85dbfcfad0d4460395b3a0ac5c2ddc19"
format_stamp: "Formatted at 2026-06-20 06:10:45 on dist-test-slave-9hm5"
server_key: "353eaed6c5ab4c30ab33fa53992a8531"
server_key_iv: "3f78aac51504a78957696ef96f2bb368"
server_key_version: "encryptionkey@0"
I20260620 06:10:45.475399 29845 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781935603120653-11672-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781935603120653-11672-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781935603120653-11672-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:10:45.498575 29845 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:10:45.501403 29845 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:10:45.501610 29845 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:10:45.502208 29845 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 06:10:45.503039 29845 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 06:10:45.503110 29845 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 06:10:45.503181 29845 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 06:10:45.503228 29845 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 06:10:45.512131 29845 rpc_server.cc:307] RPC server started. Bound to: 127.11.102.1:46215
I20260620 06:10:45.512153 29973 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.102.1:46215 every 8 connection(s)
I20260620 06:10:45.512996 29845 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781935603120653-11672-0/minicluster-data/ts-0/data/info.pb
I20260620 06:10:45.515903 11672 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9HMp43/build/debug/bin/kudu as pid 29845
I20260620 06:10:45.516047 11672 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781935603120653-11672-0/minicluster-data/ts-0/wal/instance
I20260620 06:10:45.516345 11672 external_mini_cluster.cc:1428] Setting key 1f1484fcef81661a8119d079b300af1b
Jun 20 06:10:45 dist-test-slave-9hm5 krb5kdc[29738](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935845, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.102.1@KRBTEST.COM for kudu/127.11.102.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.11.102.2@KRBTEST.COM; defaulting to no policy
I20260620 06:10:45.527277 29822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:10:45.514746 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.102.1:60489 (local address 127.11.102.62:42775)
0620 06:10:45.514875 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0620 06:10:45.514878 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 06:10:45.515648 (+ 770us) server_negotiation.cc:407] Connection header received
0620 06:10:45.517202 (+ 1554us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:10:45.517206 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:10:45.517262 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:10:45.517339 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:10:45.518587 (+ 1248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:10:45.519296 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:10:45.520523 (+ 1227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:10:45.520965 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:10:45.521086 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:10:45.523501 (+ 2415us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:10:45.523505 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:10:45.523508 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:10:45.523539 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:10:45.525224 (+ 1685us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:10:45.525719 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:10:45.525725 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:10:45.525728 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:10:45.525812 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:10:45.526151 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:10:45.526156 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:10:45.526157 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:10:45.526331 (+ 174us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:10:45.526447 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0620 06:10:45.526985 (+ 538us) server_negotiation.cc:299] Negotiation successful
0620 06:10:45.527105 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.102.2@KRBTEST.COM" created.
I20260620 06:10:45.528301 29976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:10:45.515064 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.102.62:42775 (local address 127.11.102.1:60489)
0620 06:10:45.515493 (+ 429us) negotiation.cc:107] Waiting for socket to connect
0620 06:10:45.515535 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0620 06:10:45.516995 (+ 1460us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:10:45.517511 (+ 516us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:10:45.517517 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:10:45.517963 (+ 446us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:10:45.518429 (+ 466us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:10:45.518441 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:10:45.519441 (+ 1000us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:10:45.519444 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:10:45.520394 (+ 950us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:10:45.520404 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:10:45.521111 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:10:45.521114 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:10:45.521254 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:10:45.521945 (+ 691us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:10:45.521961 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:10:45.523331 (+ 1370us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:10:45.525350 (+ 2019us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:10:45.525356 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:10:45.525370 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:10:45.525609 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:10:45.525917 (+ 308us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:10:45.525920 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:10:45.525924 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:10:45.526054 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:10:45.526445 (+ 391us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:10:45.526452 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:10:45.526749 (+ 297us) client_negotiation.cc:769] Sending connection context
0620 06:10:45.526945 (+ 196us) client_negotiation.cc:240] Negotiation successful
0620 06:10:45.527252 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":245,"thread_start_us":115,"threads_started":1}
I20260620 06:10:45.529495 29974 heartbeater.cc:347] Connected to a master server at 127.11.102.62:42775
I20260620 06:10:45.529734 29974 heartbeater.cc:464] Registering TS with master...
I20260620 06:10:45.530207 29974 heartbeater.cc:511] Master 127.11.102.62:42775 requested a full tablet report, sending...
I20260620 06:10:45.531859 29783 ts_manager.cc:194] Registered new tserver with Master: 85dbfcfad0d4460395b3a0ac5c2ddc19 (127.11.102.1:46215)
I20260620 06:10:45.533548 29783 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.102.1@KRBTEST.COM'} at 127.11.102.1:60489
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.102.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781935603120653-11672-0/krb5kdc/kudu_127.11.102.2.keytab.
Entry for principal kudu/127.11.102.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781935603120653-11672-0/krb5kdc/kudu_127.11.102.2.keytab.
I20260620 06:10:45.542867 11672 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9HMp43/build/debug/bin/kudu
/tmp/dist-test-task9HMp43/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781935603120653-11672-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781935603120653-11672-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781935603120653-11672-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781935603120653-11672-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.102.2:0
--local_ip_for_outbound_sockets=127.11.102.2
--webserver_interface=127.11.102.2
--webserver_port=0
--tserver_master_addrs=127.11.102.62:42775
--keytab_file=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781935603120653-11672-0/krb5kdc/kudu_127.11.102.2.keytab
--principal=kudu/127.11.102.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.102.20:40897
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781935603120653-11672-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781935603120653-11672-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781935603120653-11672-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 06:10:45.654708 29979 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:10:45.655004 29979 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:10:45.655082 29979 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:10:45.659263 29979 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:10:45.659350 29979 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:10:45.659446 29979 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.102.2
I20260620 06:10:45.664754 29979 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.102.20:40897
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781935603120653-11672-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781935603120653-11672-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781935603120653-11672-0/krb5kdc/kudu_127.11.102.2.keytab
--principal=kudu/127.11.102.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.102.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781935603120653-11672-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.11.102.2
--webserver_port=0
--tserver_master_addrs=127.11.102.62:42775
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29979
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.102.2
--log_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781935603120653-11672-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 05:28:48 UTC on ccycloud.quasar-fixhms.root.comops.site
build id 4457020
I20260620 06:10:45.665771 29979 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:10:45.666749 29979 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 06:10:45.676074 29985 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:10:45.676054 29987 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:10:45.676070 29984 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:10:45.676285 29979 server_base.cc:1034] running on GCE node
I20260620 06:10:45.676807 29979 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:10:45.677425 29979 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:10:45.678695 29979 hybrid_clock.cc:648] HybridClock initialized: now 1781935845678658 us; error 44 us; skew 500 ppm
Jun 20 06:10:45 dist-test-slave-9hm5 krb5kdc[29738](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935845, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.102.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 06:10:45.684355 29979 init.cc:377] Logged in from keytab as kudu/127.11.102.2@KRBTEST.COM (short username kudu)
I20260620 06:10:45.685230 29979 webserver.cc:460] Webserver started at http://127.11.102.2:37719/ using document root <none> and password file <none>
I20260620 06:10:45.685808 29979 fs_manager.cc:359] Metadata directory not provided
I20260620 06:10:45.685900 29979 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:10:45.686132 29979 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 06:10:45.688277 29979 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781935603120653-11672-0/minicluster-data/ts-1/data/instance:
uuid: "2650501b84ef4a5a93b4d194cbe634ad"
format_stamp: "Formatted at 2026-06-20 06:10:45 on dist-test-slave-9hm5"
server_key: "c813146f6828dfd5175e9a08ec1e28f1"
server_key_iv: "1d7f72d014080a6a1aa6b832a506a12b"
server_key_version: "encryptionkey@0"
I20260620 06:10:45.688818 29979 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781935603120653-11672-0/minicluster-data/ts-1/wal/instance:
uuid: "2650501b84ef4a5a93b4d194cbe634ad"
format_stamp: "Formatted at 2026-06-20 06:10:45 on dist-test-slave-9hm5"
server_key: "c813146f6828dfd5175e9a08ec1e28f1"
server_key_iv: "1d7f72d014080a6a1aa6b832a506a12b"
server_key_version: "encryptionkey@0"
I20260620 06:10:45.692328 29979 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 06:10:45.694607 29994 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:10:45.695767 29979 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 06:10:45.695890 29979 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781935603120653-11672-0/minicluster-data/ts-1/data,/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781935603120653-11672-0/minicluster-data/ts-1/wal
uuid: "2650501b84ef4a5a93b4d194cbe634ad"
format_stamp: "Formatted at 2026-06-20 06:10:45 on dist-test-slave-9hm5"
server_key: "c813146f6828dfd5175e9a08ec1e28f1"
server_key_iv: "1d7f72d014080a6a1aa6b832a506a12b"
server_key_version: "encryptionkey@0"
I20260620 06:10:45.695996 29979 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781935603120653-11672-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781935603120653-11672-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781935603120653-11672-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:10:45.705458 29979 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:10:45.708717 29979 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:10:45.708909 29979 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:10:45.709460 29979 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 06:10:45.710373 29979 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 06:10:45.710443 29979 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 06:10:45.710510 29979 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 06:10:45.710567 29979 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 06:10:45.719139 29979 rpc_server.cc:307] RPC server started. Bound to: 127.11.102.2:46125
I20260620 06:10:45.719161 30107 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.102.2:46125 every 8 connection(s)
I20260620 06:10:45.720047 29979 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781935603120653-11672-0/minicluster-data/ts-1/data/info.pb
I20260620 06:10:45.728767 11672 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9HMp43/build/debug/bin/kudu as pid 29979
I20260620 06:10:45.728899 11672 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781935603120653-11672-0/minicluster-data/ts-1/wal/instance
Jun 20 06:10:45 dist-test-slave-9hm5 krb5kdc[29738](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935845, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.102.2@KRBTEST.COM for kudu/127.11.102.62@KRBTEST.COM
I20260620 06:10:45.729252 11672 external_mini_cluster.cc:1428] Setting key e2393e454202f5ff3d74b022c63402db
I20260620 06:10:45.733608 29822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:10:45.721765 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.102.2:40895 (local address 127.11.102.62:42775)
0620 06:10:45.721927 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0620 06:10:45.721931 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 06:10:45.722752 (+ 821us) server_negotiation.cc:407] Connection header received
0620 06:10:45.724024 (+ 1272us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:10:45.724027 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:10:45.724080 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:10:45.724156 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:10:45.725242 (+ 1086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:10:45.725786 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:10:45.726975 (+ 1189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:10:45.727200 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:10:45.727282 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:10:45.729755 (+ 2473us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:10:45.729758 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:10:45.729760 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:10:45.729783 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:10:45.731189 (+ 1406us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:10:45.731858 (+ 669us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:10:45.731861 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:10:45.731862 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:10:45.731936 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:10:45.732368 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:10:45.732374 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:10:45.732377 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:10:45.732650 (+ 273us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:10:45.732756 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0620 06:10:45.733240 (+ 484us) server_negotiation.cc:299] Negotiation successful
0620 06:10:45.733387 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260620 06:10:45.734218 30110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:10:45.722169 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.102.62:42775 (local address 127.11.102.2:40895)
0620 06:10:45.722603 (+ 434us) negotiation.cc:107] Waiting for socket to connect
0620 06:10:45.722643 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0620 06:10:45.723836 (+ 1193us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:10:45.724317 (+ 481us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:10:45.724323 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:10:45.724661 (+ 338us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:10:45.725117 (+ 456us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:10:45.725130 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:10:45.725933 (+ 803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:10:45.725935 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:10:45.726862 (+ 927us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:10:45.726871 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:10:45.727325 (+ 454us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:10:45.727328 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:10:45.727460 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:10:45.728030 (+ 570us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:10:45.728046 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:10:45.729559 (+ 1513us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:10:45.731308 (+ 1749us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:10:45.731317 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:10:45.731336 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:10:45.731764 (+ 428us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:10:45.732036 (+ 272us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:10:45.732039 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:10:45.732041 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:10:45.732248 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:10:45.732757 (+ 509us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:10:45.732762 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:10:45.732981 (+ 219us) client_negotiation.cc:769] Sending connection context
0620 06:10:45.733140 (+ 159us) client_negotiation.cc:240] Negotiation successful
0620 06:10:45.733378 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":251,"thread_start_us":117,"threads_started":1}
I20260620 06:10:45.735759 30108 heartbeater.cc:347] Connected to a master server at 127.11.102.62:42775
I20260620 06:10:45.735997 30108 heartbeater.cc:464] Registering TS with master...
I20260620 06:10:45.736573 30108 heartbeater.cc:511] Master 127.11.102.62:42775 requested a full tablet report, sending...
I20260620 06:10:45.737756 29783 ts_manager.cc:194] Registered new tserver with Master: 2650501b84ef4a5a93b4d194cbe634ad (127.11.102.2:46125)
I20260620 06:10:45.738555 29783 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.102.2@KRBTEST.COM'} at 127.11.102.2:40895
WARNING: no policy specified for kudu/127.11.102.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.102.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.102.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781935603120653-11672-0/krb5kdc/kudu_127.11.102.3.keytab.
Entry for principal kudu/127.11.102.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781935603120653-11672-0/krb5kdc/kudu_127.11.102.3.keytab.
I20260620 06:10:45.756356 11672 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9HMp43/build/debug/bin/kudu
/tmp/dist-test-task9HMp43/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781935603120653-11672-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781935603120653-11672-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781935603120653-11672-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781935603120653-11672-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.102.3:0
--local_ip_for_outbound_sockets=127.11.102.3
--webserver_interface=127.11.102.3
--webserver_port=0
--tserver_master_addrs=127.11.102.62:42775
--keytab_file=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781935603120653-11672-0/krb5kdc/kudu_127.11.102.3.keytab
--principal=kudu/127.11.102.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.102.20:40897
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781935603120653-11672-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781935603120653-11672-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781935603120653-11672-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 06:10:45.863842 30113 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:10:45.864084 30113 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:10:45.864126 30113 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:10:45.868142 30113 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:10:45.868192 30113 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:10:45.868276 30113 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.102.3
I20260620 06:10:45.873565 30113 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.102.20:40897
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781935603120653-11672-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781935603120653-11672-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781935603120653-11672-0/krb5kdc/kudu_127.11.102.3.keytab
--principal=kudu/127.11.102.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.102.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781935603120653-11672-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.11.102.3
--webserver_port=0
--tserver_master_addrs=127.11.102.62:42775
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30113
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.102.3
--log_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781935603120653-11672-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 05:28:48 UTC on ccycloud.quasar-fixhms.root.comops.site
build id 4457020
I20260620 06:10:45.874521 30113 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:10:45.875470 30113 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 06:10:45.884905 30118 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:10:45.884969 30113 server_base.cc:1034] running on GCE node
W20260620 06:10:45.884914 30121 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:10:45.884914 30119 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:10:45.885650 30113 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:10:45.886230 30113 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:10:45.887389 30113 hybrid_clock.cc:648] HybridClock initialized: now 1781935845887365 us; error 37 us; skew 500 ppm
Jun 20 06:10:45 dist-test-slave-9hm5 krb5kdc[29738](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935845, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.102.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 06:10:45.890117 30113 init.cc:377] Logged in from keytab as kudu/127.11.102.3@KRBTEST.COM (short username kudu)
I20260620 06:10:45.890916 30113 webserver.cc:460] Webserver started at http://127.11.102.3:45837/ using document root <none> and password file <none>
I20260620 06:10:45.891428 30113 fs_manager.cc:359] Metadata directory not provided
I20260620 06:10:45.891485 30113 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:10:45.891714 30113 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 06:10:45.893659 30113 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781935603120653-11672-0/minicluster-data/ts-2/data/instance:
uuid: "df705ca221844f07b697a757b474c3b9"
format_stamp: "Formatted at 2026-06-20 06:10:45 on dist-test-slave-9hm5"
server_key: "88ca942fef86cfaf02013507fa5eb308"
server_key_iv: "2513d7ccec386ab65c75ac4eeb4817ce"
server_key_version: "encryptionkey@0"
I20260620 06:10:45.894163 30113 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781935603120653-11672-0/minicluster-data/ts-2/wal/instance:
uuid: "df705ca221844f07b697a757b474c3b9"
format_stamp: "Formatted at 2026-06-20 06:10:45 on dist-test-slave-9hm5"
server_key: "88ca942fef86cfaf02013507fa5eb308"
server_key_iv: "2513d7ccec386ab65c75ac4eeb4817ce"
server_key_version: "encryptionkey@0"
I20260620 06:10:45.897753 30113 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260620 06:10:45.900149 30128 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:10:45.901341 30113 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 06:10:45.901473 30113 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781935603120653-11672-0/minicluster-data/ts-2/data,/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781935603120653-11672-0/minicluster-data/ts-2/wal
uuid: "df705ca221844f07b697a757b474c3b9"
format_stamp: "Formatted at 2026-06-20 06:10:45 on dist-test-slave-9hm5"
server_key: "88ca942fef86cfaf02013507fa5eb308"
server_key_iv: "2513d7ccec386ab65c75ac4eeb4817ce"
server_key_version: "encryptionkey@0"
I20260620 06:10:45.901589 30113 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781935603120653-11672-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781935603120653-11672-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781935603120653-11672-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:10:45.921775 30113 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:10:45.924600 30113 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:10:45.924793 30113 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:10:45.925381 30113 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 06:10:45.926229 30113 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 06:10:45.926295 30113 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 06:10:45.926362 30113 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 06:10:45.926409 30113 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 06:10:45.934813 30113 rpc_server.cc:307] RPC server started. Bound to: 127.11.102.3:45489
I20260620 06:10:45.934834 30241 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.102.3:45489 every 8 connection(s)
I20260620 06:10:45.935741 30113 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781935603120653-11672-0/minicluster-data/ts-2/data/info.pb
I20260620 06:10:45.941900 11672 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9HMp43/build/debug/bin/kudu as pid 30113
I20260620 06:10:45.942029 11672 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781935603120653-11672-0/minicluster-data/ts-2/wal/instance
I20260620 06:10:45.942338 11672 external_mini_cluster.cc:1428] Setting key a2e0be05c5ace585282b1f2dd0749922
Jun 20 06:10:45 dist-test-slave-9hm5 krb5kdc[29738](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935845, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.102.3@KRBTEST.COM for kudu/127.11.102.62@KRBTEST.COM
I20260620 06:10:45.948982 29822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:10:45.937573 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.102.3:55125 (local address 127.11.102.62:42775)
0620 06:10:45.937715 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0620 06:10:45.937719 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 06:10:45.938534 (+ 815us) server_negotiation.cc:407] Connection header received
0620 06:10:45.939805 (+ 1271us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:10:45.939809 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:10:45.939868 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:10:45.939949 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:10:45.941072 (+ 1123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:10:45.941616 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:10:45.943021 (+ 1405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:10:45.943365 (+ 344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:10:45.943478 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:10:45.945978 (+ 2500us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:10:45.945980 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:10:45.945982 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:10:45.946007 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:10:45.947219 (+ 1212us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:10:45.947737 (+ 518us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:10:45.947740 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:10:45.947741 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:10:45.947814 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:10:45.948112 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:10:45.948114 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:10:45.948115 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:10:45.948274 (+ 159us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:10:45.948344 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0620 06:10:45.948772 (+ 428us) server_negotiation.cc:299] Negotiation successful
0620 06:10:45.948870 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260620 06:10:45.949726 30244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:10:45.937938 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.102.62:42775 (local address 127.11.102.3:55125)
0620 06:10:45.938366 (+ 428us) negotiation.cc:107] Waiting for socket to connect
0620 06:10:45.938408 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0620 06:10:45.939572 (+ 1164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:10:45.940120 (+ 548us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:10:45.940126 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:10:45.940462 (+ 336us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:10:45.940918 (+ 456us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:10:45.940930 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:10:45.941743 (+ 813us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:10:45.941747 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:10:45.942894 (+ 1147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:10:45.942903 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:10:45.943532 (+ 629us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:10:45.943536 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:10:45.943762 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:10:45.944381 (+ 619us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:10:45.944403 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:10:45.945834 (+ 1431us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:10:45.947317 (+ 1483us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:10:45.947322 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:10:45.947334 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:10:45.947569 (+ 235us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:10:45.947892 (+ 323us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:10:45.947894 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:10:45.947895 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:10:45.948030 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:10:45.948354 (+ 324us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:10:45.948359 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:10:45.948572 (+ 213us) client_negotiation.cc:769] Sending connection context
0620 06:10:45.948715 (+ 143us) client_negotiation.cc:240] Negotiation successful
0620 06:10:45.948945 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":255,"thread_start_us":122,"threads_started":1}
I20260620 06:10:45.950723 30242 heartbeater.cc:347] Connected to a master server at 127.11.102.62:42775
I20260620 06:10:45.950961 30242 heartbeater.cc:464] Registering TS with master...
I20260620 06:10:45.951421 30242 heartbeater.cc:511] Master 127.11.102.62:42775 requested a full tablet report, sending...
I20260620 06:10:45.952402 29783 ts_manager.cc:194] Registered new tserver with Master: df705ca221844f07b697a757b474c3b9 (127.11.102.3:45489)
I20260620 06:10:45.952947 29783 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.102.3@KRBTEST.COM'} at 127.11.102.3:55125
I20260620 06:10:45.956175 11672 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 06:10:45.964988 29822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:10:45.958053 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35026 (local address 127.11.102.62:42775)
0620 06:10:45.958187 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0620 06:10:45.958191 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 06:10:45.958361 (+ 170us) server_negotiation.cc:407] Connection header received
0620 06:10:45.958445 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:10:45.958447 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:10:45.958484 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:10:45.958552 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:10:45.959181 (+ 629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:10:45.959734 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:10:45.960507 (+ 773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:10:45.960718 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:10:45.960791 (+ 73us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:10:45.962256 (+ 1465us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:10:45.962258 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:10:45.962260 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:10:45.962278 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:10:45.963420 (+ 1142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:10:45.963930 (+ 510us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:10:45.963933 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:10:45.963934 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:10:45.964004 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:10:45.964256 (+ 252us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:10:45.964258 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:10:45.964259 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:10:45.964432 (+ 173us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:10:45.964539 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0620 06:10:45.964735 (+ 196us) server_negotiation.cc:299] Negotiation successful
0620 06:10:45.964829 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260620 06:10:45.967815 29783 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35026:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 06:10:45.970870 29783 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 06:10:45.985035 30253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:10:45.978857 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.102.3:45489 (local address 127.0.0.1:38662)
0620 06:10:45.979477 (+ 620us) negotiation.cc:107] Waiting for socket to connect
0620 06:10:45.979523 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0620 06:10:45.979691 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:10:45.980851 (+ 1160us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:10:45.980854 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:10:45.980873 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 06:10:45.980975 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:10:45.980981 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:10:45.982098 (+ 1117us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:10:45.982104 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:10:45.983654 (+ 1550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:10:45.983666 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:10:45.984505 (+ 839us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:10:45.984508 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:10:45.984607 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:10:45.984642 (+ 35us) client_negotiation.cc:769] Sending connection context
0620 06:10:45.984770 (+ 128us) client_negotiation.cc:240] Negotiation successful
0620 06:10:45.984889 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":452,"mutex_wait_us":160,"thread_start_us":64,"threads_started":1}
I20260620 06:10:45.985044 30254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:10:45.978972 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38662 (local address 127.11.102.3:45489)
0620 06:10:45.980417 (+ 1445us) server_negotiation.cc:206] Beginning negotiation
0620 06:10:45.980429 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 06:10:45.980456 (+ 27us) server_negotiation.cc:407] Connection header received
0620 06:10:45.980522 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:10:45.980525 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:10:45.980702 (+ 177us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:10:45.980855 (+ 153us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 06:10:45.981110 (+ 255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:10:45.981976 (+ 866us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:10:45.983777 (+ 1801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:10:45.984371 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:10:45.984499 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:10:45.984642 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0620 06:10:45.984811 (+ 169us) server_negotiation.cc:299] Negotiation successful
0620 06:10:45.984903 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1327,"thread_start_us":147,"threads_started":1}
I20260620 06:10:45.985399 30256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:10:45.979012 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.102.1:46215 (local address 127.0.0.1:52904)
0620 06:10:45.980063 (+ 1051us) negotiation.cc:107] Waiting for socket to connect
0620 06:10:45.980077 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 06:10:45.980158 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:10:45.980692 (+ 534us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:10:45.980698 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:10:45.980721 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 06:10:45.980880 (+ 159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:10:45.980888 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:10:45.982361 (+ 1473us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:10:45.982364 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:10:45.983688 (+ 1324us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:10:45.983697 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:10:45.984977 (+ 1280us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:10:45.984980 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:10:45.985058 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:10:45.985075 (+ 17us) client_negotiation.cc:769] Sending connection context
0620 06:10:45.985169 (+ 94us) client_negotiation.cc:240] Negotiation successful
0620 06:10:45.985259 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":989,"thread_start_us":113,"threads_started":1}
I20260620 06:10:45.985623 30255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:10:45.979579 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.102.2:46125 (local address 127.0.0.1:49896)
0620 06:10:45.980908 (+ 1329us) negotiation.cc:107] Waiting for socket to connect
0620 06:10:45.980921 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 06:10:45.981003 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:10:45.981446 (+ 443us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:10:45.981449 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:10:45.981471 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 06:10:45.981550 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:10:45.981555 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:10:45.982799 (+ 1244us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:10:45.982802 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:10:45.983787 (+ 985us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:10:45.983797 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:10:45.985214 (+ 1417us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:10:45.985218 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:10:45.985310 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:10:45.985335 (+ 25us) client_negotiation.cc:769] Sending connection context
0620 06:10:45.985437 (+ 102us) client_negotiation.cc:240] Negotiation successful
0620 06:10:45.985507 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1271,"thread_start_us":87,"threads_started":1}
I20260620 06:10:45.985833 30258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:10:45.979992 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49896 (local address 127.11.102.2:46125)
0620 06:10:45.980459 (+ 467us) server_negotiation.cc:206] Beginning negotiation
0620 06:10:45.980471 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 06:10:45.981022 (+ 551us) server_negotiation.cc:407] Connection header received
0620 06:10:45.981135 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:10:45.981139 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:10:45.981302 (+ 163us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:10:45.981444 (+ 142us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 06:10:45.981676 (+ 232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:10:45.982660 (+ 984us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:10:45.983911 (+ 1251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:10:45.984672 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:10:45.984797 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:10:45.984913 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0620 06:10:45.985478 (+ 565us) server_negotiation.cc:299] Negotiation successful
0620 06:10:45.985568 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":317,"thread_start_us":118,"threads_started":1}
I20260620 06:10:45.985894 30257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:10:45.979583 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52904 (local address 127.11.102.1:46215)
0620 06:10:45.980176 (+ 593us) server_negotiation.cc:206] Beginning negotiation
0620 06:10:45.980191 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 06:10:45.980215 (+ 24us) server_negotiation.cc:407] Connection header received
0620 06:10:45.980294 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:10:45.980300 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:10:45.980511 (+ 211us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:10:45.980663 (+ 152us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 06:10:45.981028 (+ 365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:10:45.982226 (+ 1198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:10:45.983820 (+ 1594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:10:45.984672 (+ 852us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:10:45.984780 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:10:45.984914 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0620 06:10:45.985667 (+ 753us) server_negotiation.cc:299] Negotiation successful
0620 06:10:45.985758 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":482,"thread_start_us":126,"threads_started":1}
I20260620 06:10:45.988308 30176 tablet_service.cc:1467] Processing CreateTablet for tablet 41de9d797d2a40edbb95485313bb118b (DEFAULT_TABLE table=test-table [id=39933e7f14f74c1aa3b749e52346f57d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 06:10:45.988536 30042 tablet_service.cc:1467] Processing CreateTablet for tablet 41de9d797d2a40edbb95485313bb118b (DEFAULT_TABLE table=test-table [id=39933e7f14f74c1aa3b749e52346f57d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 06:10:45.989398 30042 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 41de9d797d2a40edbb95485313bb118b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 06:10:45.989416 30176 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 41de9d797d2a40edbb95485313bb118b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 06:10:45.989331 29908 tablet_service.cc:1467] Processing CreateTablet for tablet 41de9d797d2a40edbb95485313bb118b (DEFAULT_TABLE table=test-table [id=39933e7f14f74c1aa3b749e52346f57d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 06:10:45.990208 29908 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 41de9d797d2a40edbb95485313bb118b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 06:10:45.996219 30259 tablet_bootstrap.cc:492] T 41de9d797d2a40edbb95485313bb118b P 2650501b84ef4a5a93b4d194cbe634ad: Bootstrap starting.
I20260620 06:10:45.997598 30261 tablet_bootstrap.cc:492] T 41de9d797d2a40edbb95485313bb118b P df705ca221844f07b697a757b474c3b9: Bootstrap starting.
I20260620 06:10:45.998612 30259 tablet_bootstrap.cc:654] T 41de9d797d2a40edbb95485313bb118b P 2650501b84ef4a5a93b4d194cbe634ad: Neither blocks nor log segments found. Creating new log.
I20260620 06:10:45.999356 30261 tablet_bootstrap.cc:654] T 41de9d797d2a40edbb95485313bb118b P df705ca221844f07b697a757b474c3b9: Neither blocks nor log segments found. Creating new log.
I20260620 06:10:45.999377 30259 log.cc:824] T 41de9d797d2a40edbb95485313bb118b P 2650501b84ef4a5a93b4d194cbe634ad: Log is configured to *not* fsync() on all Append() calls
I20260620 06:10:46.000075 30260 tablet_bootstrap.cc:492] T 41de9d797d2a40edbb95485313bb118b P 85dbfcfad0d4460395b3a0ac5c2ddc19: Bootstrap starting.
I20260620 06:10:46.000566 30261 log.cc:824] T 41de9d797d2a40edbb95485313bb118b P df705ca221844f07b697a757b474c3b9: Log is configured to *not* fsync() on all Append() calls
I20260620 06:10:46.001647 30259 tablet_bootstrap.cc:492] T 41de9d797d2a40edbb95485313bb118b P 2650501b84ef4a5a93b4d194cbe634ad: No bootstrap required, opened a new log
I20260620 06:10:46.001852 30259 ts_tablet_manager.cc:1397] T 41de9d797d2a40edbb95485313bb118b P 2650501b84ef4a5a93b4d194cbe634ad: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260620 06:10:46.002122 30260 tablet_bootstrap.cc:654] T 41de9d797d2a40edbb95485313bb118b P 85dbfcfad0d4460395b3a0ac5c2ddc19: Neither blocks nor log segments found. Creating new log.
I20260620 06:10:46.002628 30261 tablet_bootstrap.cc:492] T 41de9d797d2a40edbb95485313bb118b P df705ca221844f07b697a757b474c3b9: No bootstrap required, opened a new log
I20260620 06:10:46.002826 30261 ts_tablet_manager.cc:1397] T 41de9d797d2a40edbb95485313bb118b P df705ca221844f07b697a757b474c3b9: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260620 06:10:46.003275 30260 log.cc:824] T 41de9d797d2a40edbb95485313bb118b P 85dbfcfad0d4460395b3a0ac5c2ddc19: Log is configured to *not* fsync() on all Append() calls
I20260620 06:10:46.005192 30260 tablet_bootstrap.cc:492] T 41de9d797d2a40edbb95485313bb118b P 85dbfcfad0d4460395b3a0ac5c2ddc19: No bootstrap required, opened a new log
I20260620 06:10:46.005378 30260 ts_tablet_manager.cc:1397] T 41de9d797d2a40edbb95485313bb118b P 85dbfcfad0d4460395b3a0ac5c2ddc19: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.003s
I20260620 06:10:46.005271 30259 raft_consensus.cc:348] T 41de9d797d2a40edbb95485313bb118b P 2650501b84ef4a5a93b4d194cbe634ad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df705ca221844f07b697a757b474c3b9" member_type: VOTER last_known_addr { host: "127.11.102.3" port: 45489 } } peers { permanent_uuid: "85dbfcfad0d4460395b3a0ac5c2ddc19" member_type: VOTER last_known_addr { host: "127.11.102.1" port: 46215 } } peers { permanent_uuid: "2650501b84ef4a5a93b4d194cbe634ad" member_type: VOTER last_known_addr { host: "127.11.102.2" port: 46125 } }
I20260620 06:10:46.005584 30259 raft_consensus.cc:374] T 41de9d797d2a40edbb95485313bb118b P 2650501b84ef4a5a93b4d194cbe634ad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 06:10:46.005659 30259 raft_consensus.cc:729] T 41de9d797d2a40edbb95485313bb118b P 2650501b84ef4a5a93b4d194cbe634ad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2650501b84ef4a5a93b4d194cbe634ad, State: Initialized, Role: FOLLOWER
I20260620 06:10:46.006166 30259 consensus_queue.cc:260] T 41de9d797d2a40edbb95485313bb118b P 2650501b84ef4a5a93b4d194cbe634ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df705ca221844f07b697a757b474c3b9" member_type: VOTER last_known_addr { host: "127.11.102.3" port: 45489 } } peers { permanent_uuid: "85dbfcfad0d4460395b3a0ac5c2ddc19" member_type: VOTER last_known_addr { host: "127.11.102.1" port: 46215 } } peers { permanent_uuid: "2650501b84ef4a5a93b4d194cbe634ad" member_type: VOTER last_known_addr { host: "127.11.102.2" port: 46125 } }
I20260620 06:10:46.006517 30261 raft_consensus.cc:348] T 41de9d797d2a40edbb95485313bb118b P df705ca221844f07b697a757b474c3b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df705ca221844f07b697a757b474c3b9" member_type: VOTER last_known_addr { host: "127.11.102.3" port: 45489 } } peers { permanent_uuid: "85dbfcfad0d4460395b3a0ac5c2ddc19" member_type: VOTER last_known_addr { host: "127.11.102.1" port: 46215 } } peers { permanent_uuid: "2650501b84ef4a5a93b4d194cbe634ad" member_type: VOTER last_known_addr { host: "127.11.102.2" port: 46125 } }
I20260620 06:10:46.006774 30261 raft_consensus.cc:374] T 41de9d797d2a40edbb95485313bb118b P df705ca221844f07b697a757b474c3b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 06:10:46.006789 30108 heartbeater.cc:503] Master 127.11.102.62:42775 was elected leader, sending a full tablet report...
I20260620 06:10:46.006834 30261 raft_consensus.cc:729] T 41de9d797d2a40edbb95485313bb118b P df705ca221844f07b697a757b474c3b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df705ca221844f07b697a757b474c3b9, State: Initialized, Role: FOLLOWER
I20260620 06:10:46.007284 30261 consensus_queue.cc:260] T 41de9d797d2a40edbb95485313bb118b P df705ca221844f07b697a757b474c3b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df705ca221844f07b697a757b474c3b9" member_type: VOTER last_known_addr { host: "127.11.102.3" port: 45489 } } peers { permanent_uuid: "85dbfcfad0d4460395b3a0ac5c2ddc19" member_type: VOTER last_known_addr { host: "127.11.102.1" port: 46215 } } peers { permanent_uuid: "2650501b84ef4a5a93b4d194cbe634ad" member_type: VOTER last_known_addr { host: "127.11.102.2" port: 46125 } }
I20260620 06:10:46.008265 30261 ts_tablet_manager.cc:1428] T 41de9d797d2a40edbb95485313bb118b P df705ca221844f07b697a757b474c3b9: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260620 06:10:46.008613 30242 heartbeater.cc:503] Master 127.11.102.62:42775 was elected leader, sending a full tablet report...
I20260620 06:10:46.008977 30260 raft_consensus.cc:348] T 41de9d797d2a40edbb95485313bb118b P 85dbfcfad0d4460395b3a0ac5c2ddc19 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df705ca221844f07b697a757b474c3b9" member_type: VOTER last_known_addr { host: "127.11.102.3" port: 45489 } } peers { permanent_uuid: "85dbfcfad0d4460395b3a0ac5c2ddc19" member_type: VOTER last_known_addr { host: "127.11.102.1" port: 46215 } } peers { permanent_uuid: "2650501b84ef4a5a93b4d194cbe634ad" member_type: VOTER last_known_addr { host: "127.11.102.2" port: 46125 } }
I20260620 06:10:46.009243 30260 raft_consensus.cc:374] T 41de9d797d2a40edbb95485313bb118b P 85dbfcfad0d4460395b3a0ac5c2ddc19 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 06:10:46.009300 30260 raft_consensus.cc:729] T 41de9d797d2a40edbb95485313bb118b P 85dbfcfad0d4460395b3a0ac5c2ddc19 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85dbfcfad0d4460395b3a0ac5c2ddc19, State: Initialized, Role: FOLLOWER
I20260620 06:10:46.009755 30260 consensus_queue.cc:260] T 41de9d797d2a40edbb95485313bb118b P 85dbfcfad0d4460395b3a0ac5c2ddc19 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df705ca221844f07b697a757b474c3b9" member_type: VOTER last_known_addr { host: "127.11.102.3" port: 45489 } } peers { permanent_uuid: "85dbfcfad0d4460395b3a0ac5c2ddc19" member_type: VOTER last_known_addr { host: "127.11.102.1" port: 46215 } } peers { permanent_uuid: "2650501b84ef4a5a93b4d194cbe634ad" member_type: VOTER last_known_addr { host: "127.11.102.2" port: 46125 } }
I20260620 06:10:46.010394 29974 heartbeater.cc:503] Master 127.11.102.62:42775 was elected leader, sending a full tablet report...
I20260620 06:10:46.010700 30259 ts_tablet_manager.cc:1428] T 41de9d797d2a40edbb95485313bb118b P 2650501b84ef4a5a93b4d194cbe634ad: Time spent starting tablet: real 0.009s user 0.006s sys 0.000s
I20260620 06:10:46.010742 30260 ts_tablet_manager.cc:1428] T 41de9d797d2a40edbb95485313bb118b P 85dbfcfad0d4460395b3a0ac5c2ddc19: Time spent starting tablet: real 0.005s user 0.005s sys 0.002s
W20260620 06:10:46.014043 29975 tablet.cc:2369] T 41de9d797d2a40edbb95485313bb118b P 85dbfcfad0d4460395b3a0ac5c2ddc19: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 06:10:46.076079 30267 raft_consensus.cc:482] T 41de9d797d2a40edbb95485313bb118b P 85dbfcfad0d4460395b3a0ac5c2ddc19 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 06:10:46.076300 30267 raft_consensus.cc:504] T 41de9d797d2a40edbb95485313bb118b P 85dbfcfad0d4460395b3a0ac5c2ddc19 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df705ca221844f07b697a757b474c3b9" member_type: VOTER last_known_addr { host: "127.11.102.3" port: 45489 } } peers { permanent_uuid: "85dbfcfad0d4460395b3a0ac5c2ddc19" member_type: VOTER last_known_addr { host: "127.11.102.1" port: 46215 } } peers { permanent_uuid: "2650501b84ef4a5a93b4d194cbe634ad" member_type: VOTER last_known_addr { host: "127.11.102.2" port: 46125 } }
I20260620 06:10:46.077430 30267 leader_election.cc:288] T 41de9d797d2a40edbb95485313bb118b P 85dbfcfad0d4460395b3a0ac5c2ddc19 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df705ca221844f07b697a757b474c3b9 (127.11.102.3:45489), 2650501b84ef4a5a93b4d194cbe634ad (127.11.102.2:46125)
I20260620 06:10:46.081284 30254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:10:46.077803 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.102.1:51135 (local address 127.11.102.3:45489)
0620 06:10:46.077941 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0620 06:10:46.077946 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 06:10:46.078051 (+ 105us) server_negotiation.cc:407] Connection header received
0620 06:10:46.078214 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:10:46.078218 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:10:46.078270 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:10:46.078357 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 06:10:46.078642 (+ 285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:10:46.079242 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:10:46.080321 (+ 1079us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:10:46.080733 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:10:46.080859 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:10:46.080911 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0620 06:10:46.081098 (+ 187us) server_negotiation.cc:299] Negotiation successful
0620 06:10:46.081166 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260620 06:10:46.081305 30268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:10:46.077677 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.102.3:45489 (local address 127.11.102.1:51135)
0620 06:10:46.077968 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0620 06:10:46.077984 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 06:10:46.078093 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:10:46.078372 (+ 279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:10:46.078375 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:10:46.078407 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 06:10:46.078530 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:10:46.078536 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:10:46.079350 (+ 814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:10:46.079356 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:10:46.080192 (+ 836us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:10:46.080201 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:10:46.080854 (+ 653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:10:46.080858 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:10:46.080979 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:10:46.080998 (+ 19us) client_negotiation.cc:769] Sending connection context
0620 06:10:46.081085 (+ 87us) client_negotiation.cc:240] Negotiation successful
0620 06:10:46.081154 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":212,"spinlock_wait_cycles":8320,"thread_start_us":91,"threads_started":1}
I20260620 06:10:46.081784 30269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:10:46.077677 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.102.2:46125 (local address 127.11.102.1:39997)
0620 06:10:46.078257 (+ 580us) negotiation.cc:107] Waiting for socket to connect
0620 06:10:46.078268 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 06:10:46.078406 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:10:46.078678 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:10:46.078681 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:10:46.078695 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 06:10:46.078812 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:10:46.078818 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:10:46.079737 (+ 919us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:10:46.079741 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:10:46.080528 (+ 787us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:10:46.080536 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:10:46.081375 (+ 839us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:10:46.081379 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:10:46.081474 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:10:46.081495 (+ 21us) client_negotiation.cc:769] Sending connection context
0620 06:10:46.081578 (+ 83us) client_negotiation.cc:240] Negotiation successful
0620 06:10:46.081651 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":517,"mutex_wait_us":244,"spinlock_wait_cycles":2304,"thread_start_us":134,"threads_started":1}
I20260620 06:10:46.081804 30258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:10:46.077804 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.102.1:39997 (local address 127.11.102.2:46125)
0620 06:10:46.077979 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0620 06:10:46.077984 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 06:10:46.078360 (+ 376us) server_negotiation.cc:407] Connection header received
0620 06:10:46.078528 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:10:46.078531 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:10:46.078586 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:10:46.078670 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 06:10:46.078924 (+ 254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:10:46.079601 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:10:46.080653 (+ 1052us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:10:46.081163 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:10:46.081271 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:10:46.081315 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0620 06:10:46.081611 (+ 296us) server_negotiation.cc:299] Negotiation successful
0620 06:10:46.081684 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":94}
I20260620 06:10:46.082369 30062 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "41de9d797d2a40edbb95485313bb118b" candidate_uuid: "85dbfcfad0d4460395b3a0ac5c2ddc19" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2650501b84ef4a5a93b4d194cbe634ad" is_pre_election: true
I20260620 06:10:46.082370 30196 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "41de9d797d2a40edbb95485313bb118b" candidate_uuid: "85dbfcfad0d4460395b3a0ac5c2ddc19" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df705ca221844f07b697a757b474c3b9" is_pre_election: true
I20260620 06:10:46.082667 30196 raft_consensus.cc:2454] T 41de9d797d2a40edbb95485313bb118b P df705ca221844f07b697a757b474c3b9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 85dbfcfad0d4460395b3a0ac5c2ddc19 in term 0.
I20260620 06:10:46.082667 30062 raft_consensus.cc:2454] T 41de9d797d2a40edbb95485313bb118b P 2650501b84ef4a5a93b4d194cbe634ad [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 85dbfcfad0d4460395b3a0ac5c2ddc19 in term 0.
I20260620 06:10:46.083125 29863 leader_election.cc:302] T 41de9d797d2a40edbb95485313bb118b P 85dbfcfad0d4460395b3a0ac5c2ddc19 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85dbfcfad0d4460395b3a0ac5c2ddc19, df705ca221844f07b697a757b474c3b9; no voters:
I20260620 06:10:46.083370 30267 raft_consensus.cc:2781] T 41de9d797d2a40edbb95485313bb118b P 85dbfcfad0d4460395b3a0ac5c2ddc19 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 06:10:46.083472 30267 raft_consensus.cc:482] T 41de9d797d2a40edbb95485313bb118b P 85dbfcfad0d4460395b3a0ac5c2ddc19 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 06:10:46.083530 30267 raft_consensus.cc:3037] T 41de9d797d2a40edbb95485313bb118b P 85dbfcfad0d4460395b3a0ac5c2ddc19 [term 0 FOLLOWER]: Advancing to term 1
I20260620 06:10:46.084553 30267 raft_consensus.cc:504] T 41de9d797d2a40edbb95485313bb118b P 85dbfcfad0d4460395b3a0ac5c2ddc19 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df705ca221844f07b697a757b474c3b9" member_type: VOTER last_known_addr { host: "127.11.102.3" port: 45489 } } peers { permanent_uuid: "85dbfcfad0d4460395b3a0ac5c2ddc19" member_type: VOTER last_known_addr { host: "127.11.102.1" port: 46215 } } peers { permanent_uuid: "2650501b84ef4a5a93b4d194cbe634ad" member_type: VOTER last_known_addr { host: "127.11.102.2" port: 46125 } }
I20260620 06:10:46.084898 30267 leader_election.cc:288] T 41de9d797d2a40edbb95485313bb118b P 85dbfcfad0d4460395b3a0ac5c2ddc19 [CANDIDATE]: Term 1 election: Requested vote from peers df705ca221844f07b697a757b474c3b9 (127.11.102.3:45489), 2650501b84ef4a5a93b4d194cbe634ad (127.11.102.2:46125)
I20260620 06:10:46.085215 30196 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "41de9d797d2a40edbb95485313bb118b" candidate_uuid: "85dbfcfad0d4460395b3a0ac5c2ddc19" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df705ca221844f07b697a757b474c3b9"
I20260620 06:10:46.085270 30062 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "41de9d797d2a40edbb95485313bb118b" candidate_uuid: "85dbfcfad0d4460395b3a0ac5c2ddc19" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2650501b84ef4a5a93b4d194cbe634ad"
I20260620 06:10:46.085330 30196 raft_consensus.cc:3037] T 41de9d797d2a40edbb95485313bb118b P df705ca221844f07b697a757b474c3b9 [term 0 FOLLOWER]: Advancing to term 1
I20260620 06:10:46.085410 30062 raft_consensus.cc:3037] T 41de9d797d2a40edbb95485313bb118b P 2650501b84ef4a5a93b4d194cbe634ad [term 0 FOLLOWER]: Advancing to term 1
I20260620 06:10:46.086342 30196 raft_consensus.cc:2454] T 41de9d797d2a40edbb95485313bb118b P df705ca221844f07b697a757b474c3b9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 85dbfcfad0d4460395b3a0ac5c2ddc19 in term 1.
I20260620 06:10:46.086463 30062 raft_consensus.cc:2454] T 41de9d797d2a40edbb95485313bb118b P 2650501b84ef4a5a93b4d194cbe634ad [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 85dbfcfad0d4460395b3a0ac5c2ddc19 in term 1.
I20260620 06:10:46.086691 29863 leader_election.cc:302] T 41de9d797d2a40edbb95485313bb118b P 85dbfcfad0d4460395b3a0ac5c2ddc19 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85dbfcfad0d4460395b3a0ac5c2ddc19, df705ca221844f07b697a757b474c3b9; no voters:
I20260620 06:10:46.086849 30267 raft_consensus.cc:2781] T 41de9d797d2a40edbb95485313bb118b P 85dbfcfad0d4460395b3a0ac5c2ddc19 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 06:10:46.087096 30267 raft_consensus.cc:686] T 41de9d797d2a40edbb95485313bb118b P 85dbfcfad0d4460395b3a0ac5c2ddc19 [term 1 LEADER]: Becoming Leader. State: Replica: 85dbfcfad0d4460395b3a0ac5c2ddc19, State: Running, Role: LEADER
I20260620 06:10:46.087411 30267 consensus_queue.cc:237] T 41de9d797d2a40edbb95485313bb118b P 85dbfcfad0d4460395b3a0ac5c2ddc19 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df705ca221844f07b697a757b474c3b9" member_type: VOTER last_known_addr { host: "127.11.102.3" port: 45489 } } peers { permanent_uuid: "85dbfcfad0d4460395b3a0ac5c2ddc19" member_type: VOTER last_known_addr { host: "127.11.102.1" port: 46215 } } peers { permanent_uuid: "2650501b84ef4a5a93b4d194cbe634ad" member_type: VOTER last_known_addr { host: "127.11.102.2" port: 46125 } }
I20260620 06:10:46.090603 29782 catalog_manager.cc:5495] T 41de9d797d2a40edbb95485313bb118b P 85dbfcfad0d4460395b3a0ac5c2ddc19 reported cstate change: term changed from 0 to 1, leader changed from <none> to 85dbfcfad0d4460395b3a0ac5c2ddc19 (127.11.102.1). New cstate: current_term: 1 leader_uuid: "85dbfcfad0d4460395b3a0ac5c2ddc19" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df705ca221844f07b697a757b474c3b9" member_type: VOTER last_known_addr { host: "127.11.102.3" port: 45489 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "85dbfcfad0d4460395b3a0ac5c2ddc19" member_type: VOTER last_known_addr { host: "127.11.102.1" port: 46215 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2650501b84ef4a5a93b4d194cbe634ad" member_type: VOTER last_known_addr { host: "127.11.102.2" port: 46125 } health_report { overall_health: UNKNOWN } } }
I20260620 06:10:46.105811 30257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:10:46.102483 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52918 (local address 127.11.102.1:46215)
0620 06:10:46.102595 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0620 06:10:46.102598 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 06:10:46.102611 (+ 13us) server_negotiation.cc:407] Connection header received
0620 06:10:46.102752 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:10:46.102754 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:10:46.102800 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:10:46.102888 (+ 88us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 06:10:46.103106 (+ 218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:10:46.103587 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:10:46.104518 (+ 931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:10:46.104716 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:10:46.104786 (+ 70us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:10:46.105074 (+ 288us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 06:10:46.105493 (+ 419us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 06:10:46.105604 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0620 06:10:46.105682 (+ 78us) server_negotiation.cc:299] Negotiation successful
0620 06:10:46.105713 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260620 06:10:46.111343 30196 raft_consensus.cc:1261] T 41de9d797d2a40edbb95485313bb118b P df705ca221844f07b697a757b474c3b9 [term 1 FOLLOWER]: Refusing update from remote peer 85dbfcfad0d4460395b3a0ac5c2ddc19: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 06:10:46.111343 30062 raft_consensus.cc:1261] T 41de9d797d2a40edbb95485313bb118b P 2650501b84ef4a5a93b4d194cbe634ad [term 1 FOLLOWER]: Refusing update from remote peer 85dbfcfad0d4460395b3a0ac5c2ddc19: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 06:10:46.112087 30270 consensus_queue.cc:1035] T 41de9d797d2a40edbb95485313bb118b P 85dbfcfad0d4460395b3a0ac5c2ddc19 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df705ca221844f07b697a757b474c3b9" member_type: VOTER last_known_addr { host: "127.11.102.3" port: 45489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 06:10:46.112267 30267 consensus_queue.cc:1035] T 41de9d797d2a40edbb95485313bb118b P 85dbfcfad0d4460395b3a0ac5c2ddc19 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2650501b84ef4a5a93b4d194cbe634ad" member_type: VOTER last_known_addr { host: "127.11.102.2" port: 46125 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 06:10:46.120019 30275 mvcc.cc:205] Tried to move back new op lower bound from 7298809225665683456 to 7298809225576865792. Current Snapshot: MvccSnapshot[applied={T|T < 7298809225665683456}]
I20260620 06:10:46.127388 29779 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35026:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 06:10:46.127568 29779 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35026:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 06:10:46.129848 29779 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 958f6124650044afa74f9609ea59faad: Sending DeleteTablet for 3 replicas of tablet 41de9d797d2a40edbb95485313bb118b
I20260620 06:10:46.130584 30176 tablet_service.cc:1514] Processing DeleteTablet for tablet 41de9d797d2a40edbb95485313bb118b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 06:10:46 UTC) from {username='kudu', principal='kudu/127.11.102.62@KRBTEST.COM'} at 127.0.0.1:38662
I20260620 06:10:46.130568 29908 tablet_service.cc:1514] Processing DeleteTablet for tablet 41de9d797d2a40edbb95485313bb118b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 06:10:46 UTC) from {username='kudu', principal='kudu/127.11.102.62@KRBTEST.COM'} at 127.0.0.1:52904
I20260620 06:10:46.131069 30283 tablet_replica.cc:330] T 41de9d797d2a40edbb95485313bb118b P 85dbfcfad0d4460395b3a0ac5c2ddc19: stopping tablet replica
I20260620 06:10:46.131325 30042 tablet_service.cc:1514] Processing DeleteTablet for tablet 41de9d797d2a40edbb95485313bb118b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 06:10:46 UTC) from {username='kudu', principal='kudu/127.11.102.62@KRBTEST.COM'} at 127.0.0.1:49896
I20260620 06:10:46.131412 30283 raft_consensus.cc:2229] T 41de9d797d2a40edbb95485313bb118b P 85dbfcfad0d4460395b3a0ac5c2ddc19 [term 1 LEADER]: Raft consensus shutting down.
I20260620 06:10:46.131769 11672 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9HMp43/build/debug/bin/kudu with pid 29845
I20260620 06:10:46.131798 30285 tablet_replica.cc:330] T 41de9d797d2a40edbb95485313bb118b P 2650501b84ef4a5a93b4d194cbe634ad: stopping tablet replica
I20260620 06:10:46.132513 30284 tablet_replica.cc:330] T 41de9d797d2a40edbb95485313bb118b P df705ca221844f07b697a757b474c3b9: stopping tablet replica
I20260620 06:10:46.132767 30284 raft_consensus.cc:2229] T 41de9d797d2a40edbb95485313bb118b P df705ca221844f07b697a757b474c3b9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 06:10:46.133045 30284 raft_consensus.cc:2258] T 41de9d797d2a40edbb95485313bb118b P df705ca221844f07b697a757b474c3b9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 06:10:46.134543 30285 raft_consensus.cc:2229] T 41de9d797d2a40edbb95485313bb118b P 2650501b84ef4a5a93b4d194cbe634ad [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 06:10:46.134577 30284 ts_tablet_manager.cc:1905] T 41de9d797d2a40edbb95485313bb118b P df705ca221844f07b697a757b474c3b9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 06:10:46.134819 30285 raft_consensus.cc:2258] T 41de9d797d2a40edbb95485313bb118b P 2650501b84ef4a5a93b4d194cbe634ad [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 06:10:46.136111 30285 ts_tablet_manager.cc:1905] T 41de9d797d2a40edbb95485313bb118b P 2650501b84ef4a5a93b4d194cbe634ad: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 06:10:46.137959 30284 ts_tablet_manager.cc:1918] T 41de9d797d2a40edbb95485313bb118b P df705ca221844f07b697a757b474c3b9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 06:10:46.138067 30284 log.cc:1196] T 41de9d797d2a40edbb95485313bb118b P df705ca221844f07b697a757b474c3b9: Deleting WAL directory at /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781935603120653-11672-0/minicluster-data/ts-2/wal/wals/41de9d797d2a40edbb95485313bb118b
I20260620 06:10:46.138089 30285 ts_tablet_manager.cc:1918] T 41de9d797d2a40edbb95485313bb118b P 2650501b84ef4a5a93b4d194cbe634ad: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 06:10:46.138180 30285 log.cc:1196] T 41de9d797d2a40edbb95485313bb118b P 2650501b84ef4a5a93b4d194cbe634ad: Deleting WAL directory at /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781935603120653-11672-0/minicluster-data/ts-1/wal/wals/41de9d797d2a40edbb95485313bb118b
I20260620 06:10:46.138404 30284 ts_tablet_manager.cc:1939] T 41de9d797d2a40edbb95485313bb118b P df705ca221844f07b697a757b474c3b9: Deleting consensus metadata
I20260620 06:10:46.138497 30285 ts_tablet_manager.cc:1939] T 41de9d797d2a40edbb95485313bb118b P 2650501b84ef4a5a93b4d194cbe634ad: Deleting consensus metadata
I20260620 06:10:46.139582 29769 catalog_manager.cc:4841] TS df705ca221844f07b697a757b474c3b9 (127.11.102.3:45489): tablet 41de9d797d2a40edbb95485313bb118b (table test-table [id=39933e7f14f74c1aa3b749e52346f57d]) successfully deleted
I20260620 06:10:46.139771 11672 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9HMp43/build/debug/bin/kudu with pid 29979
I20260620 06:10:46.139822 29768 catalog_manager.cc:4841] TS 2650501b84ef4a5a93b4d194cbe634ad (127.11.102.2:46125): tablet 41de9d797d2a40edbb95485313bb118b (table test-table [id=39933e7f14f74c1aa3b749e52346f57d]) successfully deleted
W20260620 06:10:46.139896 29767 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.11.102.1:46215 (error 108)
I20260620 06:10:46.144734 30255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:10:46.144493 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.102.1:46215 (local address 127.0.0.1:52928)
0620 06:10:46.144612 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0620 06:10:46.144668 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.102.1:46215: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260620 06:10:46.144882 29767 catalog_manager.cc:4584] TS 85dbfcfad0d4460395b3a0ac5c2ddc19 (127.11.102.1:46215): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 41de9d797d2a40edbb95485313bb118b: Network error: Client connection negotiation failed: client connection to 127.11.102.1:46215: connect: Connection refused (error 111)
I20260620 06:10:46.145228 11672 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9HMp43/build/debug/bin/kudu with pid 30113
I20260620 06:10:46.151216 11672 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9HMp43/build/debug/bin/kudu with pid 29751
2026-06-20T06:10:46Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (2246 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-06-20T06:10:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T06:10:46Z Disabled control of system clock
I20260620 06:10:46.181092 11672 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9HMp43/build/debug/bin/kudu
/tmp/dist-test-task9HMp43/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781935603120653-11672-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781935603120653-11672-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781935603120653-11672-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781935603120653-11672-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.102.62:33531
--webserver_interface=127.11.102.62
--webserver_port=0
--builtin_ntp_servers=127.11.102.20:42887
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.102.62:33531
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260620 06:10:46.286404 30290 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:10:46.286652 30290 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:10:46.286685 30290 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:10:46.290426 30290 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 06:10:46.290469 30290 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:10:46.290484 30290 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:10:46.290500 30290 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 06:10:46.290514 30290 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 06:10:46.294955 30290 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.102.20:42887
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781935603120653-11672-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781935603120653-11672-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.102.62:33531
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.102.62:33531
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781935603120653-11672-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.102.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30290
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781935603120653-11672-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 05:28:48 UTC on ccycloud.quasar-fixhms.root.comops.site
build id 4457020
I20260620 06:10:46.295953 30290 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:10:46.297189 30290 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 06:10:46.305757 30290 server_base.cc:1034] running on GCE node
W20260620 06:10:46.305809 30298 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:10:46.305763 30295 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:10:46.305779 30296 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:10:46.306429 30290 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:10:46.307476 30290 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:10:46.308658 30290 hybrid_clock.cc:648] HybridClock initialized: now 1781935846308622 us; error 44 us; skew 500 ppm
I20260620 06:10:46.310341 30290 webserver.cc:460] Webserver started at http://127.11.102.62:40147/ using document root <none> and password file <none>
I20260620 06:10:46.310879 30290 fs_manager.cc:359] Metadata directory not provided
I20260620 06:10:46.310968 30290 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:10:46.311192 30290 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 06:10:46.313149 30290 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781935603120653-11672-0/minicluster-data/master-0/data/instance:
uuid: "48630a39279645f8a4eec0a6c561ee67"
format_stamp: "Formatted at 2026-06-20 06:10:46 on dist-test-slave-9hm5"
server_key: "727d7717855f3d2b659ef3c6f237da45"
server_key_iv: "ca052e60f4c8fea82fde49322c079202"
server_key_version: "encryptionkey@0"
I20260620 06:10:46.313690 30290 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781935603120653-11672-0/minicluster-data/master-0/wal/instance:
uuid: "48630a39279645f8a4eec0a6c561ee67"
format_stamp: "Formatted at 2026-06-20 06:10:46 on dist-test-slave-9hm5"
server_key: "727d7717855f3d2b659ef3c6f237da45"
server_key_iv: "ca052e60f4c8fea82fde49322c079202"
server_key_version: "encryptionkey@0"
I20260620 06:10:46.317251 30290 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260620 06:10:46.319422 30304 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:10:46.320616 30290 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 06:10:46.320750 30290 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781935603120653-11672-0/minicluster-data/master-0/data,/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781935603120653-11672-0/minicluster-data/master-0/wal
uuid: "48630a39279645f8a4eec0a6c561ee67"
format_stamp: "Formatted at 2026-06-20 06:10:46 on dist-test-slave-9hm5"
server_key: "727d7717855f3d2b659ef3c6f237da45"
server_key_iv: "ca052e60f4c8fea82fde49322c079202"
server_key_version: "encryptionkey@0"
I20260620 06:10:46.320858 30290 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781935603120653-11672-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781935603120653-11672-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781935603120653-11672-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:10:46.337327 30290 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:10:46.337952 30290 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:10:46.338140 30290 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:10:46.344874 30290 rpc_server.cc:307] RPC server started. Bound to: 127.11.102.62:33531
I20260620 06:10:46.344884 30356 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.102.62:33531 every 8 connection(s)
I20260620 06:10:46.345757 30290 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781935603120653-11672-0/minicluster-data/master-0/data/info.pb
I20260620 06:10:46.346091 11672 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9HMp43/build/debug/bin/kudu as pid 30290
I20260620 06:10:46.346186 11672 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781935603120653-11672-0/minicluster-data/master-0/wal/instance
I20260620 06:10:46.346532 11672 external_mini_cluster.cc:1428] Setting key 58575d3daf7517014fb4d9ecd81df06f
I20260620 06:10:46.349119 30357 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 06:10:46.354450 30357 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 48630a39279645f8a4eec0a6c561ee67: Bootstrap starting.
I20260620 06:10:46.355662 30360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:10:46.347968 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55700 (local address 127.11.102.62:33531)
0620 06:10:46.348569 (+ 601us) server_negotiation.cc:206] Beginning negotiation
0620 06:10:46.348582 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 06:10:46.348615 (+ 33us) server_negotiation.cc:407] Connection header received
0620 06:10:46.349288 (+ 673us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:10:46.349306 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:10:46.349683 (+ 377us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:10:46.350028 (+ 345us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:10:46.350428 (+ 400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:10:46.351386 (+ 958us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:10:46.352343 (+ 957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:10:46.352805 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:10:46.352989 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:10:46.353854 (+ 865us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:10:46.353859 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:10:46.353872 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 06:10:46.354251 (+ 379us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:10:46.354306 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 06:10:46.354325 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:10:46.354431 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0620 06:10:46.354586 (+ 155us) server_negotiation.cc:299] Negotiation successful
0620 06:10:46.354837 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":278,"thread_start_us":137,"threads_started":1}
I20260620 06:10:46.356870 30357 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 48630a39279645f8a4eec0a6c561ee67: Neither blocks nor log segments found. Creating new log.
I20260620 06:10:46.357638 30357 log.cc:824] T 00000000000000000000000000000000 P 48630a39279645f8a4eec0a6c561ee67: Log is configured to *not* fsync() on all Append() calls
I20260620 06:10:46.359473 30357 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 48630a39279645f8a4eec0a6c561ee67: No bootstrap required, opened a new log
I20260620 06:10:46.363036 30357 raft_consensus.cc:348] T 00000000000000000000000000000000 P 48630a39279645f8a4eec0a6c561ee67 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "48630a39279645f8a4eec0a6c561ee67" member_type: VOTER last_known_addr { host: "127.11.102.62" port: 33531 } }
I20260620 06:10:46.363246 30357 raft_consensus.cc:374] T 00000000000000000000000000000000 P 48630a39279645f8a4eec0a6c561ee67 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 06:10:46.363282 30357 raft_consensus.cc:729] T 00000000000000000000000000000000 P 48630a39279645f8a4eec0a6c561ee67 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 48630a39279645f8a4eec0a6c561ee67, State: Initialized, Role: FOLLOWER
I20260620 06:10:46.363687 30357 consensus_queue.cc:260] T 00000000000000000000000000000000 P 48630a39279645f8a4eec0a6c561ee67 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "48630a39279645f8a4eec0a6c561ee67" member_type: VOTER last_known_addr { host: "127.11.102.62" port: 33531 } }
I20260620 06:10:46.363806 30357 raft_consensus.cc:388] T 00000000000000000000000000000000 P 48630a39279645f8a4eec0a6c561ee67 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 06:10:46.363850 30357 raft_consensus.cc:482] T 00000000000000000000000000000000 P 48630a39279645f8a4eec0a6c561ee67 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 06:10:46.363919 30357 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 48630a39279645f8a4eec0a6c561ee67 [term 0 FOLLOWER]: Advancing to term 1
I20260620 06:10:46.364717 30357 raft_consensus.cc:504] T 00000000000000000000000000000000 P 48630a39279645f8a4eec0a6c561ee67 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "48630a39279645f8a4eec0a6c561ee67" member_type: VOTER last_known_addr { host: "127.11.102.62" port: 33531 } }
I20260620 06:10:46.364997 30357 leader_election.cc:302] T 00000000000000000000000000000000 P 48630a39279645f8a4eec0a6c561ee67 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 48630a39279645f8a4eec0a6c561ee67; no voters:
I20260620 06:10:46.365249 30357 leader_election.cc:288] T 00000000000000000000000000000000 P 48630a39279645f8a4eec0a6c561ee67 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 06:10:46.365373 30362 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 48630a39279645f8a4eec0a6c561ee67 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 06:10:46.365626 30362 raft_consensus.cc:686] T 00000000000000000000000000000000 P 48630a39279645f8a4eec0a6c561ee67 [term 1 LEADER]: Becoming Leader. State: Replica: 48630a39279645f8a4eec0a6c561ee67, State: Running, Role: LEADER
I20260620 06:10:46.365929 30362 consensus_queue.cc:237] T 00000000000000000000000000000000 P 48630a39279645f8a4eec0a6c561ee67 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "48630a39279645f8a4eec0a6c561ee67" member_type: VOTER last_known_addr { host: "127.11.102.62" port: 33531 } }
I20260620 06:10:46.366514 30357 sys_catalog.cc:564] T 00000000000000000000000000000000 P 48630a39279645f8a4eec0a6c561ee67 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 06:10:46.368165 30364 sys_catalog.cc:455] T 00000000000000000000000000000000 P 48630a39279645f8a4eec0a6c561ee67 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 48630a39279645f8a4eec0a6c561ee67. Latest consensus state: current_term: 1 leader_uuid: "48630a39279645f8a4eec0a6c561ee67" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "48630a39279645f8a4eec0a6c561ee67" member_type: VOTER last_known_addr { host: "127.11.102.62" port: 33531 } } }
I20260620 06:10:46.368312 30364 sys_catalog.cc:458] T 00000000000000000000000000000000 P 48630a39279645f8a4eec0a6c561ee67 [sys.catalog]: This master's current role is: LEADER
I20260620 06:10:46.368607 30363 sys_catalog.cc:455] T 00000000000000000000000000000000 P 48630a39279645f8a4eec0a6c561ee67 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "48630a39279645f8a4eec0a6c561ee67" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "48630a39279645f8a4eec0a6c561ee67" member_type: VOTER last_known_addr { host: "127.11.102.62" port: 33531 } } }
I20260620 06:10:46.368700 30363 sys_catalog.cc:458] T 00000000000000000000000000000000 P 48630a39279645f8a4eec0a6c561ee67 [sys.catalog]: This master's current role is: LEADER
W20260620 06:10:46.371238 30378 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 48630a39279645f8a4eec0a6c561ee67: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 06:10:46.371330 30378 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 06:10:46.371407 30375 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 06:10:46.372149 30375 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 06:10:46.377475 30375 catalog_manager.cc:1347] Generated new cluster ID: e34dd5d69cdd4c66940d1cd92c4dd0ce
I20260620 06:10:46.377544 30375 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 06:10:46.391453 30375 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 06:10:46.392707 30375 catalog_manager.cc:1504] Loading token signing keys...
I20260620 06:10:46.400343 30375 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 48630a39279645f8a4eec0a6c561ee67: Generated new TSK 0
I20260620 06:10:46.401023 30375 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 06:10:46.409616 11672 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9HMp43/build/debug/bin/kudu
/tmp/dist-test-task9HMp43/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781935603120653-11672-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781935603120653-11672-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781935603120653-11672-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781935603120653-11672-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.102.1:0
--local_ip_for_outbound_sockets=127.11.102.1
--webserver_interface=127.11.102.1
--webserver_port=0
--tserver_master_addrs=127.11.102.62:33531
--builtin_ntp_servers=127.11.102.20:42887
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260620 06:10:46.520716 30381 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:10:46.520964 30381 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:10:46.521006 30381 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:10:46.525146 30381 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:10:46.525200 30381 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:10:46.525295 30381 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.102.1
I20260620 06:10:46.529958 30381 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.102.20:42887
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781935603120653-11672-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781935603120653-11672-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.102.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781935603120653-11672-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.11.102.1
--webserver_port=0
--tserver_master_addrs=127.11.102.62:33531
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30381
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.102.1
--log_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781935603120653-11672-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 05:28:48 UTC on ccycloud.quasar-fixhms.root.comops.site
build id 4457020
I20260620 06:10:46.530982 30381 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:10:46.532392 30381 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 06:10:46.541548 30386 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:10:46.541548 30389 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:10:46.541625 30387 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:10:46.541647 30381 server_base.cc:1034] running on GCE node
I20260620 06:10:46.542181 30381 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:10:46.542819 30381 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:10:46.544009 30381 hybrid_clock.cc:648] HybridClock initialized: now 1781935846543979 us; error 50 us; skew 500 ppm
I20260620 06:10:46.545835 30381 webserver.cc:460] Webserver started at http://127.11.102.1:38631/ using document root <none> and password file <none>
I20260620 06:10:46.546389 30381 fs_manager.cc:359] Metadata directory not provided
I20260620 06:10:46.546486 30381 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:10:46.546717 30381 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 06:10:46.548744 30381 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781935603120653-11672-0/minicluster-data/ts-0/data/instance:
uuid: "cfd4b9d8635344bda1af0be90e9fa821"
format_stamp: "Formatted at 2026-06-20 06:10:46 on dist-test-slave-9hm5"
server_key: "b64da8f9eb54d06444cd1289ae368d58"
server_key_iv: "0915f541d9e299d7de5ca39b97f70597"
server_key_version: "encryptionkey@0"
I20260620 06:10:46.549310 30381 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781935603120653-11672-0/minicluster-data/ts-0/wal/instance:
uuid: "cfd4b9d8635344bda1af0be90e9fa821"
format_stamp: "Formatted at 2026-06-20 06:10:46 on dist-test-slave-9hm5"
server_key: "b64da8f9eb54d06444cd1289ae368d58"
server_key_iv: "0915f541d9e299d7de5ca39b97f70597"
server_key_version: "encryptionkey@0"
I20260620 06:10:46.552845 30381 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 06:10:46.555136 30395 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:10:46.556440 30381 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 06:10:46.556571 30381 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781935603120653-11672-0/minicluster-data/ts-0/data,/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781935603120653-11672-0/minicluster-data/ts-0/wal
uuid: "cfd4b9d8635344bda1af0be90e9fa821"
format_stamp: "Formatted at 2026-06-20 06:10:46 on dist-test-slave-9hm5"
server_key: "b64da8f9eb54d06444cd1289ae368d58"
server_key_iv: "0915f541d9e299d7de5ca39b97f70597"
server_key_version: "encryptionkey@0"
I20260620 06:10:46.556676 30381 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781935603120653-11672-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781935603120653-11672-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781935603120653-11672-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:10:46.573495 30381 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:10:46.574112 30381 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:10:46.574292 30381 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:10:46.574836 30381 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 06:10:46.575820 30381 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 06:10:46.575888 30381 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 06:10:46.575961 30381 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 06:10:46.576014 30381 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 06:10:46.585501 30381 rpc_server.cc:307] RPC server started. Bound to: 127.11.102.1:46053
I20260620 06:10:46.585533 30508 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.102.1:46053 every 8 connection(s)
I20260620 06:10:46.586436 30381 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781935603120653-11672-0/minicluster-data/ts-0/data/info.pb
I20260620 06:10:46.595960 30360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:10:46.588366 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.102.1:39475 (local address 127.11.102.62:33531)
0620 06:10:46.588544 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0620 06:10:46.588549 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 06:10:46.589753 (+ 1204us) server_negotiation.cc:407] Connection header received
0620 06:10:46.591160 (+ 1407us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:10:46.591164 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:10:46.591208 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:10:46.591289 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:10:46.592282 (+ 993us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:10:46.592815 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:10:46.593936 (+ 1121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:10:46.594168 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:10:46.594266 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:10:46.595291 (+ 1025us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:10:46.595294 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:10:46.595296 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 06:10:46.595309 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:10:46.595331 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 06:10:46.595341 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:10:46.595415 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0620 06:10:46.595572 (+ 157us) server_negotiation.cc:299] Negotiation successful
0620 06:10:46.595772 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260620 06:10:46.596026 11672 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9HMp43/build/debug/bin/kudu as pid 30381
I20260620 06:10:46.596148 11672 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781935603120653-11672-0/minicluster-data/ts-0/wal/instance
I20260620 06:10:46.596477 11672 external_mini_cluster.cc:1428] Setting key 9c6782d3c17efa4e6ee738a3841ca772
I20260620 06:10:46.596630 30511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:10:46.588809 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.102.62:33531 (local address 127.11.102.1:39475)
0620 06:10:46.589572 (+ 763us) negotiation.cc:107] Waiting for socket to connect
0620 06:10:46.589626 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0620 06:10:46.590947 (+ 1321us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:10:46.591488 (+ 541us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:10:46.591495 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:10:46.591581 (+ 86us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:10:46.592136 (+ 555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:10:46.592150 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:10:46.592927 (+ 777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:10:46.592929 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:10:46.593798 (+ 869us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:10:46.593807 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:10:46.594303 (+ 496us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:10:46.594306 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:10:46.594466 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:10:46.595047 (+ 581us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 06:10:46.595049 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:10:46.595087 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 06:10:46.595144 (+ 57us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:10:46.595438 (+ 294us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:10:46.595444 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:10:46.595448 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 06:10:46.595539 (+ 91us) client_negotiation.cc:240] Negotiation successful
0620 06:10:46.595802 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":268,"thread_start_us":109,"threads_started":1}
I20260620 06:10:46.597841 30509 heartbeater.cc:347] Connected to a master server at 127.11.102.62:33531
I20260620 06:10:46.598116 30509 heartbeater.cc:464] Registering TS with master...
I20260620 06:10:46.598716 30509 heartbeater.cc:511] Master 127.11.102.62:33531 requested a full tablet report, sending...
I20260620 06:10:46.598790 11672 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9HMp43/build/debug/bin/kudu
/tmp/dist-test-task9HMp43/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781935603120653-11672-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781935603120653-11672-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781935603120653-11672-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781935603120653-11672-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.102.2:0
--local_ip_for_outbound_sockets=127.11.102.2
--webserver_interface=127.11.102.2
--webserver_port=0
--tserver_master_addrs=127.11.102.62:33531
--builtin_ntp_servers=127.11.102.20:42887
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260620 06:10:46.600296 30320 ts_manager.cc:194] Registered new tserver with Master: cfd4b9d8635344bda1af0be90e9fa821 (127.11.102.1:46053)
I20260620 06:10:46.601634 30320 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.11.102.1:39475
W20260620 06:10:46.705736 30512 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:10:46.705971 30512 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:10:46.706007 30512 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:10:46.709837 30512 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:10:46.709877 30512 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:10:46.709932 30512 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.102.2
I20260620 06:10:46.714464 30512 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.102.20:42887
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781935603120653-11672-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781935603120653-11672-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.102.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781935603120653-11672-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.11.102.2
--webserver_port=0
--tserver_master_addrs=127.11.102.62:33531
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30512
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.102.2
--log_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781935603120653-11672-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 05:28:48 UTC on ccycloud.quasar-fixhms.root.comops.site
build id 4457020
I20260620 06:10:46.715364 30512 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:10:46.716634 30512 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 06:10:46.725726 30518 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:10:46.725757 30517 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:10:46.725731 30520 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:10:46.726146 30512 server_base.cc:1034] running on GCE node
I20260620 06:10:46.726581 30512 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:10:46.727210 30512 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:10:46.728389 30512 hybrid_clock.cc:648] HybridClock initialized: now 1781935846728369 us; error 38 us; skew 500 ppm
I20260620 06:10:46.730099 30512 webserver.cc:460] Webserver started at http://127.11.102.2:39185/ using document root <none> and password file <none>
I20260620 06:10:46.730644 30512 fs_manager.cc:359] Metadata directory not provided
I20260620 06:10:46.730731 30512 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:10:46.730938 30512 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 06:10:46.732926 30512 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781935603120653-11672-0/minicluster-data/ts-1/data/instance:
uuid: "16ee41aaa52c43f581f1e5018f99ee95"
format_stamp: "Formatted at 2026-06-20 06:10:46 on dist-test-slave-9hm5"
server_key: "1aeba7c1b92e3e5d7e2d1fdac9ef9640"
server_key_iv: "e632391b111f881190f7fa3af417ed4b"
server_key_version: "encryptionkey@0"
I20260620 06:10:46.733462 30512 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781935603120653-11672-0/minicluster-data/ts-1/wal/instance:
uuid: "16ee41aaa52c43f581f1e5018f99ee95"
format_stamp: "Formatted at 2026-06-20 06:10:46 on dist-test-slave-9hm5"
server_key: "1aeba7c1b92e3e5d7e2d1fdac9ef9640"
server_key_iv: "e632391b111f881190f7fa3af417ed4b"
server_key_version: "encryptionkey@0"
I20260620 06:10:46.736896 30512 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 06:10:46.739847 30526 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:10:46.741264 30512 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 06:10:46.741377 30512 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781935603120653-11672-0/minicluster-data/ts-1/data,/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781935603120653-11672-0/minicluster-data/ts-1/wal
uuid: "16ee41aaa52c43f581f1e5018f99ee95"
format_stamp: "Formatted at 2026-06-20 06:10:46 on dist-test-slave-9hm5"
server_key: "1aeba7c1b92e3e5d7e2d1fdac9ef9640"
server_key_iv: "e632391b111f881190f7fa3af417ed4b"
server_key_version: "encryptionkey@0"
I20260620 06:10:46.741496 30512 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781935603120653-11672-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781935603120653-11672-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781935603120653-11672-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:10:46.787518 30512 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:10:46.788286 30512 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:10:46.788486 30512 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:10:46.789080 30512 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 06:10:46.790007 30512 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 06:10:46.790052 30512 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 06:10:46.790119 30512 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 06:10:46.790158 30512 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 06:10:46.799201 30512 rpc_server.cc:307] RPC server started. Bound to: 127.11.102.2:40143
I20260620 06:10:46.799325 30639 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.102.2:40143 every 8 connection(s)
I20260620 06:10:46.800180 30512 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781935603120653-11672-0/minicluster-data/ts-1/data/info.pb
I20260620 06:10:46.805794 11672 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9HMp43/build/debug/bin/kudu as pid 30512
I20260620 06:10:46.805940 11672 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781935603120653-11672-0/minicluster-data/ts-1/wal/instance
I20260620 06:10:46.806298 11672 external_mini_cluster.cc:1428] Setting key 30c18deb93041477540735f0e3c5bc6a
I20260620 06:10:46.808184 11672 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9HMp43/build/debug/bin/kudu
/tmp/dist-test-task9HMp43/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781935603120653-11672-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781935603120653-11672-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781935603120653-11672-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781935603120653-11672-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.102.3:0
--local_ip_for_outbound_sockets=127.11.102.3
--webserver_interface=127.11.102.3
--webserver_port=0
--tserver_master_addrs=127.11.102.62:33531
--builtin_ntp_servers=127.11.102.20:42887
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260620 06:10:46.809417 30360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:10:46.801955 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.102.2:49437 (local address 127.11.102.62:33531)
0620 06:10:46.802086 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0620 06:10:46.802089 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 06:10:46.803086 (+ 997us) server_negotiation.cc:407] Connection header received
0620 06:10:46.804536 (+ 1450us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:10:46.804540 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:10:46.804597 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:10:46.804697 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:10:46.805624 (+ 927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:10:46.806289 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:10:46.807443 (+ 1154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:10:46.807710 (+ 267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:10:46.807819 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:10:46.808892 (+ 1073us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:10:46.808896 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:10:46.808899 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 06:10:46.808912 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:10:46.808939 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 06:10:46.808947 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:10:46.809043 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0620 06:10:46.809209 (+ 166us) server_negotiation.cc:299] Negotiation successful
0620 06:10:46.809271 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260620 06:10:46.810173 30642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:10:46.802283 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.102.62:33531 (local address 127.11.102.2:49437)
0620 06:10:46.802930 (+ 647us) negotiation.cc:107] Waiting for socket to connect
0620 06:10:46.802970 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0620 06:10:46.804323 (+ 1353us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:10:46.804866 (+ 543us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:10:46.804873 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:10:46.804979 (+ 106us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:10:46.805446 (+ 467us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:10:46.805458 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:10:46.806411 (+ 953us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:10:46.806415 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:10:46.807293 (+ 878us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:10:46.807301 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:10:46.807855 (+ 554us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:10:46.807859 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:10:46.808025 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:10:46.808632 (+ 607us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 06:10:46.808633 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:10:46.808671 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 06:10:46.808731 (+ 60us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:10:46.809057 (+ 326us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:10:46.809067 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:10:46.809072 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 06:10:46.809197 (+ 125us) client_negotiation.cc:240] Negotiation successful
0620 06:10:46.809459 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":262,"thread_start_us":110,"threads_started":1}
I20260620 06:10:46.811233 30640 heartbeater.cc:347] Connected to a master server at 127.11.102.62:33531
I20260620 06:10:46.811467 30640 heartbeater.cc:464] Registering TS with master...
I20260620 06:10:46.812089 30640 heartbeater.cc:511] Master 127.11.102.62:33531 requested a full tablet report, sending...
I20260620 06:10:46.813174 30320 ts_manager.cc:194] Registered new tserver with Master: 16ee41aaa52c43f581f1e5018f99ee95 (127.11.102.2:40143)
I20260620 06:10:46.813742 30320 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.11.102.2:49437
W20260620 06:10:46.916250 30643 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:10:46.916488 30643 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:10:46.916525 30643 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:10:46.920500 30643 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:10:46.920547 30643 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:10:46.920626 30643 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.102.3
I20260620 06:10:46.925405 30643 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.102.20:42887
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781935603120653-11672-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781935603120653-11672-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.102.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781935603120653-11672-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.11.102.3
--webserver_port=0
--tserver_master_addrs=127.11.102.62:33531
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30643
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.102.3
--log_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781935603120653-11672-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 05:28:48 UTC on ccycloud.quasar-fixhms.root.comops.site
build id 4457020
I20260620 06:10:46.926363 30643 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:10:46.927726 30643 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 06:10:46.937052 30648 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:10:46.937192 30651 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:10:46.937088 30649 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:10:46.937089 30643 server_base.cc:1034] running on GCE node
I20260620 06:10:46.937606 30643 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:10:46.938198 30643 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:10:46.939355 30643 hybrid_clock.cc:648] HybridClock initialized: now 1781935846939319 us; error 47 us; skew 500 ppm
I20260620 06:10:46.941061 30643 webserver.cc:460] Webserver started at http://127.11.102.3:46459/ using document root <none> and password file <none>
I20260620 06:10:46.941602 30643 fs_manager.cc:359] Metadata directory not provided
I20260620 06:10:46.941661 30643 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:10:46.941864 30643 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 06:10:46.943851 30643 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781935603120653-11672-0/minicluster-data/ts-2/data/instance:
uuid: "00991649baf741c18408069f4e7cbc76"
format_stamp: "Formatted at 2026-06-20 06:10:46 on dist-test-slave-9hm5"
server_key: "c6de35e5d836c0389894643679441767"
server_key_iv: "035b26f0a48d787a2db526ea80a24c68"
server_key_version: "encryptionkey@0"
I20260620 06:10:46.944394 30643 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781935603120653-11672-0/minicluster-data/ts-2/wal/instance:
uuid: "00991649baf741c18408069f4e7cbc76"
format_stamp: "Formatted at 2026-06-20 06:10:46 on dist-test-slave-9hm5"
server_key: "c6de35e5d836c0389894643679441767"
server_key_iv: "035b26f0a48d787a2db526ea80a24c68"
server_key_version: "encryptionkey@0"
I20260620 06:10:46.947841 30643 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 06:10:46.950166 30657 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:10:46.951282 30643 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 06:10:46.951411 30643 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781935603120653-11672-0/minicluster-data/ts-2/data,/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781935603120653-11672-0/minicluster-data/ts-2/wal
uuid: "00991649baf741c18408069f4e7cbc76"
format_stamp: "Formatted at 2026-06-20 06:10:46 on dist-test-slave-9hm5"
server_key: "c6de35e5d836c0389894643679441767"
server_key_iv: "035b26f0a48d787a2db526ea80a24c68"
server_key_version: "encryptionkey@0"
I20260620 06:10:46.951519 30643 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781935603120653-11672-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781935603120653-11672-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781935603120653-11672-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:10:46.981950 30643 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:10:46.982676 30643 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:10:46.982870 30643 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:10:46.983464 30643 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 06:10:46.984479 30643 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 06:10:46.984549 30643 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 06:10:46.984619 30643 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 06:10:46.984668 30643 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 06:10:46.993196 30643 rpc_server.cc:307] RPC server started. Bound to: 127.11.102.3:34161
I20260620 06:10:46.993268 30770 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.102.3:34161 every 8 connection(s)
I20260620 06:10:46.994099 30643 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781935603120653-11672-0/minicluster-data/ts-2/data/info.pb
I20260620 06:10:46.994640 11672 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9HMp43/build/debug/bin/kudu as pid 30643
I20260620 06:10:46.994760 11672 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781935603120653-11672-0/minicluster-data/ts-2/wal/instance
I20260620 06:10:46.995101 11672 external_mini_cluster.cc:1428] Setting key ecf41fcff21cea12b2be4e1c536e3d4d
I20260620 06:10:47.003986 30360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:10:46.996264 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.102.3:46009 (local address 127.11.102.62:33531)
0620 06:10:46.996407 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0620 06:10:46.996414 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 06:10:46.997663 (+ 1249us) server_negotiation.cc:407] Connection header received
0620 06:10:46.998957 (+ 1294us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:10:46.998960 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:10:46.999003 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:10:46.999069 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:10:46.999957 (+ 888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:10:47.000837 (+ 880us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:10:47.002116 (+ 1279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:10:47.002340 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:10:47.002422 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:10:47.003534 (+ 1112us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:10:47.003537 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:10:47.003539 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 06:10:47.003551 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:10:47.003569 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 06:10:47.003576 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:10:47.003677 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0620 06:10:47.003819 (+ 142us) server_negotiation.cc:299] Negotiation successful
0620 06:10:47.003870 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260620 06:10:47.004884 30773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:10:46.996813 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.102.62:33531 (local address 127.11.102.3:46009)
0620 06:10:46.997547 (+ 734us) negotiation.cc:107] Waiting for socket to connect
0620 06:10:46.997586 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0620 06:10:46.998783 (+ 1197us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:10:46.999209 (+ 426us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:10:46.999215 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:10:46.999293 (+ 78us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:10:46.999835 (+ 542us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:10:46.999849 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:10:47.000955 (+ 1106us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:10:47.000961 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:10:47.002001 (+ 1040us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:10:47.002010 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:10:47.002494 (+ 484us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:10:47.002497 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:10:47.002642 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:10:47.003281 (+ 639us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 06:10:47.003283 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:10:47.003323 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 06:10:47.003387 (+ 64us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:10:47.003692 (+ 305us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:10:47.003699 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:10:47.003703 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 06:10:47.003802 (+ 99us) client_negotiation.cc:240] Negotiation successful
0620 06:10:47.004027 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":256,"thread_start_us":99,"threads_started":1}
I20260620 06:10:47.005960 30771 heartbeater.cc:347] Connected to a master server at 127.11.102.62:33531
I20260620 06:10:47.006156 30771 heartbeater.cc:464] Registering TS with master...
I20260620 06:10:47.006608 30771 heartbeater.cc:511] Master 127.11.102.62:33531 requested a full tablet report, sending...
I20260620 06:10:47.007652 30320 ts_manager.cc:194] Registered new tserver with Master: 00991649baf741c18408069f4e7cbc76 (127.11.102.3:34161)
I20260620 06:10:47.008438 30320 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.11.102.3:46009
I20260620 06:10:47.009163 11672 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 06:10:47.014658 30360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:10:47.011007 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55710 (local address 127.11.102.62:33531)
0620 06:10:47.011131 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0620 06:10:47.011135 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 06:10:47.011320 (+ 185us) server_negotiation.cc:407] Connection header received
0620 06:10:47.011412 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:10:47.011414 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:10:47.011447 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:10:47.011527 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:10:47.011855 (+ 328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:10:47.012327 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:10:47.013163 (+ 836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:10:47.013365 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:10:47.013443 (+ 78us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:10:47.014260 (+ 817us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:10:47.014262 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:10:47.014264 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 06:10:47.014274 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:10:47.014289 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 06:10:47.014296 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:10:47.014379 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0620 06:10:47.014533 (+ 154us) server_negotiation.cc:299] Negotiation successful
0620 06:10:47.014561 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260620 06:10:47.017256 30320 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55710:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 06:10:47.020172 30320 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 06:10:47.033979 30782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:10:47.028115 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.102.1:46053 (local address 127.0.0.1:39694)
0620 06:10:47.028455 (+ 340us) negotiation.cc:107] Waiting for socket to connect
0620 06:10:47.028506 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0620 06:10:47.028621 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:10:47.029625 (+ 1004us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:10:47.029631 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:10:47.029655 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 06:10:47.029805 (+ 150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:10:47.029815 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:10:47.031105 (+ 1290us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:10:47.031111 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:10:47.032241 (+ 1130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:10:47.032250 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:10:47.033435 (+ 1185us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:10:47.033439 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:10:47.033556 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:10:47.033600 (+ 44us) client_negotiation.cc:769] Sending connection context
0620 06:10:47.033702 (+ 102us) client_negotiation.cc:240] Negotiation successful
0620 06:10:47.033819 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":159,"mutex_wait_us":1,"thread_start_us":72,"threads_started":1}
I20260620 06:10:47.034198 30787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:10:47.029314 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51382 (local address 127.11.102.3:34161)
0620 06:10:47.029755 (+ 441us) server_negotiation.cc:206] Beginning negotiation
0620 06:10:47.029765 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 06:10:47.029783 (+ 18us) server_negotiation.cc:407] Connection header received
0620 06:10:47.029841 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:10:47.029845 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:10:47.029984 (+ 139us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:10:47.030202 (+ 218us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 06:10:47.030577 (+ 375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:10:47.031720 (+ 1143us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:10:47.032945 (+ 1225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:10:47.033591 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:10:47.033706 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:10:47.033863 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0620 06:10:47.033925 (+ 62us) server_negotiation.cc:299] Negotiation successful
0620 06:10:47.034035 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":361,"thread_start_us":144,"threads_started":1}
I20260620 06:10:47.034349 30783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:10:47.028326 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.102.2:40143 (local address 127.0.0.1:51518)
0620 06:10:47.028545 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0620 06:10:47.028559 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 06:10:47.028642 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:10:47.030107 (+ 1465us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:10:47.030110 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:10:47.030137 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 06:10:47.030250 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:10:47.030257 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:10:47.031819 (+ 1562us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:10:47.031822 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:10:47.032908 (+ 1086us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:10:47.032927 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:10:47.033958 (+ 1031us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:10:47.033961 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:10:47.034054 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:10:47.034081 (+ 27us) client_negotiation.cc:769] Sending connection context
0620 06:10:47.034140 (+ 59us) client_negotiation.cc:240] Negotiation successful
0620 06:10:47.034207 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":160,"mutex_wait_us":4,"spinlock_wait_cycles":1408,"thread_start_us":105,"threads_started":1}
I20260620 06:10:47.034469 30785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:10:47.028775 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39694 (local address 127.11.102.1:46053)
0620 06:10:47.029088 (+ 313us) server_negotiation.cc:206] Beginning negotiation
0620 06:10:47.029102 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 06:10:47.029128 (+ 26us) server_negotiation.cc:407] Connection header received
0620 06:10:47.029205 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:10:47.029210 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:10:47.029418 (+ 208us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:10:47.029973 (+ 555us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 06:10:47.030116 (+ 143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:10:47.030999 (+ 883us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:10:47.032516 (+ 1517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:10:47.033312 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:10:47.033415 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:10:47.033605 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0620 06:10:47.034171 (+ 566us) server_negotiation.cc:299] Negotiation successful
0620 06:10:47.034271 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":219,"thread_start_us":137,"threads_started":1}
I20260620 06:10:47.034976 30786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:10:47.029324 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51518 (local address 127.11.102.2:40143)
0620 06:10:47.029643 (+ 319us) server_negotiation.cc:206] Beginning negotiation
0620 06:10:47.029656 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 06:10:47.029679 (+ 23us) server_negotiation.cc:407] Connection header received
0620 06:10:47.029752 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:10:47.029756 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:10:47.029935 (+ 179us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:10:47.030394 (+ 459us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 06:10:47.030515 (+ 121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:10:47.031709 (+ 1194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:10:47.033047 (+ 1338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:10:47.033817 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:10:47.034508 (+ 691us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:10:47.034684 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0620 06:10:47.034735 (+ 51us) server_negotiation.cc:299] Negotiation successful
0620 06:10:47.034822 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":219,"thread_start_us":110,"threads_started":1}
I20260620 06:10:47.035657 30784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:10:47.028795 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.102.3:34161 (local address 127.0.0.1:51382)
0620 06:10:47.029017 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0620 06:10:47.029044 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0620 06:10:47.029126 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:10:47.030116 (+ 990us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:10:47.030119 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:10:47.030138 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 06:10:47.030406 (+ 268us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:10:47.030413 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:10:47.031868 (+ 1455us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:10:47.031870 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:10:47.032807 (+ 937us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:10:47.032818 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:10:47.033716 (+ 898us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:10:47.033719 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:10:47.033803 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:10:47.033828 (+ 25us) client_negotiation.cc:769] Sending connection context
0620 06:10:47.033883 (+ 55us) client_negotiation.cc:240] Negotiation successful
0620 06:10:47.033960 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":159,"mutex_wait_us":170,"spinlock_wait_cycles":1920,"thread_start_us":106,"threads_started":1}
I20260620 06:10:47.037545 30443 tablet_service.cc:1467] Processing CreateTablet for tablet 8a3245cbce6b4de392e5645a1a28f5e5 (DEFAULT_TABLE table=test-table [id=f25b5f4c6ff644fd982c6e65227ffdfe]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 06:10:47.037539 30574 tablet_service.cc:1467] Processing CreateTablet for tablet 8a3245cbce6b4de392e5645a1a28f5e5 (DEFAULT_TABLE table=test-table [id=f25b5f4c6ff644fd982c6e65227ffdfe]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 06:10:47.037963 30705 tablet_service.cc:1467] Processing CreateTablet for tablet 8a3245cbce6b4de392e5645a1a28f5e5 (DEFAULT_TABLE table=test-table [id=f25b5f4c6ff644fd982c6e65227ffdfe]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 06:10:47.038635 30574 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8a3245cbce6b4de392e5645a1a28f5e5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 06:10:47.038635 30443 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8a3245cbce6b4de392e5645a1a28f5e5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 06:10:47.038868 30705 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8a3245cbce6b4de392e5645a1a28f5e5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 06:10:47.045581 30788 tablet_bootstrap.cc:492] T 8a3245cbce6b4de392e5645a1a28f5e5 P 00991649baf741c18408069f4e7cbc76: Bootstrap starting.
I20260620 06:10:47.046211 30789 tablet_bootstrap.cc:492] T 8a3245cbce6b4de392e5645a1a28f5e5 P 16ee41aaa52c43f581f1e5018f99ee95: Bootstrap starting.
I20260620 06:10:47.047574 30788 tablet_bootstrap.cc:654] T 8a3245cbce6b4de392e5645a1a28f5e5 P 00991649baf741c18408069f4e7cbc76: Neither blocks nor log segments found. Creating new log.
I20260620 06:10:47.047986 30790 tablet_bootstrap.cc:492] T 8a3245cbce6b4de392e5645a1a28f5e5 P cfd4b9d8635344bda1af0be90e9fa821: Bootstrap starting.
I20260620 06:10:47.048156 30789 tablet_bootstrap.cc:654] T 8a3245cbce6b4de392e5645a1a28f5e5 P 16ee41aaa52c43f581f1e5018f99ee95: Neither blocks nor log segments found. Creating new log.
I20260620 06:10:47.048470 30788 log.cc:824] T 8a3245cbce6b4de392e5645a1a28f5e5 P 00991649baf741c18408069f4e7cbc76: Log is configured to *not* fsync() on all Append() calls
I20260620 06:10:47.048895 30789 log.cc:824] T 8a3245cbce6b4de392e5645a1a28f5e5 P 16ee41aaa52c43f581f1e5018f99ee95: Log is configured to *not* fsync() on all Append() calls
I20260620 06:10:47.049997 30790 tablet_bootstrap.cc:654] T 8a3245cbce6b4de392e5645a1a28f5e5 P cfd4b9d8635344bda1af0be90e9fa821: Neither blocks nor log segments found. Creating new log.
I20260620 06:10:47.050786 30789 tablet_bootstrap.cc:492] T 8a3245cbce6b4de392e5645a1a28f5e5 P 16ee41aaa52c43f581f1e5018f99ee95: No bootstrap required, opened a new log
I20260620 06:10:47.050762 30790 log.cc:824] T 8a3245cbce6b4de392e5645a1a28f5e5 P cfd4b9d8635344bda1af0be90e9fa821: Log is configured to *not* fsync() on all Append() calls
I20260620 06:10:47.050921 30788 tablet_bootstrap.cc:492] T 8a3245cbce6b4de392e5645a1a28f5e5 P 00991649baf741c18408069f4e7cbc76: No bootstrap required, opened a new log
I20260620 06:10:47.051007 30789 ts_tablet_manager.cc:1397] T 8a3245cbce6b4de392e5645a1a28f5e5 P 16ee41aaa52c43f581f1e5018f99ee95: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260620 06:10:47.051211 30788 ts_tablet_manager.cc:1397] T 8a3245cbce6b4de392e5645a1a28f5e5 P 00991649baf741c18408069f4e7cbc76: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.005s
I20260620 06:10:47.054080 30789 raft_consensus.cc:348] T 8a3245cbce6b4de392e5645a1a28f5e5 P 16ee41aaa52c43f581f1e5018f99ee95 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfd4b9d8635344bda1af0be90e9fa821" member_type: VOTER last_known_addr { host: "127.11.102.1" port: 46053 } } peers { permanent_uuid: "16ee41aaa52c43f581f1e5018f99ee95" member_type: VOTER last_known_addr { host: "127.11.102.2" port: 40143 } } peers { permanent_uuid: "00991649baf741c18408069f4e7cbc76" member_type: VOTER last_known_addr { host: "127.11.102.3" port: 34161 } }
I20260620 06:10:47.054280 30789 raft_consensus.cc:374] T 8a3245cbce6b4de392e5645a1a28f5e5 P 16ee41aaa52c43f581f1e5018f99ee95 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 06:10:47.054131 30788 raft_consensus.cc:348] T 8a3245cbce6b4de392e5645a1a28f5e5 P 00991649baf741c18408069f4e7cbc76 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfd4b9d8635344bda1af0be90e9fa821" member_type: VOTER last_known_addr { host: "127.11.102.1" port: 46053 } } peers { permanent_uuid: "16ee41aaa52c43f581f1e5018f99ee95" member_type: VOTER last_known_addr { host: "127.11.102.2" port: 40143 } } peers { permanent_uuid: "00991649baf741c18408069f4e7cbc76" member_type: VOTER last_known_addr { host: "127.11.102.3" port: 34161 } }
I20260620 06:10:47.054339 30789 raft_consensus.cc:729] T 8a3245cbce6b4de392e5645a1a28f5e5 P 16ee41aaa52c43f581f1e5018f99ee95 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 16ee41aaa52c43f581f1e5018f99ee95, State: Initialized, Role: FOLLOWER
I20260620 06:10:47.054374 30788 raft_consensus.cc:374] T 8a3245cbce6b4de392e5645a1a28f5e5 P 00991649baf741c18408069f4e7cbc76 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 06:10:47.054440 30788 raft_consensus.cc:729] T 8a3245cbce6b4de392e5645a1a28f5e5 P 00991649baf741c18408069f4e7cbc76 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00991649baf741c18408069f4e7cbc76, State: Initialized, Role: FOLLOWER
I20260620 06:10:47.054791 30789 consensus_queue.cc:260] T 8a3245cbce6b4de392e5645a1a28f5e5 P 16ee41aaa52c43f581f1e5018f99ee95 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfd4b9d8635344bda1af0be90e9fa821" member_type: VOTER last_known_addr { host: "127.11.102.1" port: 46053 } } peers { permanent_uuid: "16ee41aaa52c43f581f1e5018f99ee95" member_type: VOTER last_known_addr { host: "127.11.102.2" port: 40143 } } peers { permanent_uuid: "00991649baf741c18408069f4e7cbc76" member_type: VOTER last_known_addr { host: "127.11.102.3" port: 34161 } }
I20260620 06:10:47.054839 30788 consensus_queue.cc:260] T 8a3245cbce6b4de392e5645a1a28f5e5 P 00991649baf741c18408069f4e7cbc76 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfd4b9d8635344bda1af0be90e9fa821" member_type: VOTER last_known_addr { host: "127.11.102.1" port: 46053 } } peers { permanent_uuid: "16ee41aaa52c43f581f1e5018f99ee95" member_type: VOTER last_known_addr { host: "127.11.102.2" port: 40143 } } peers { permanent_uuid: "00991649baf741c18408069f4e7cbc76" member_type: VOTER last_known_addr { host: "127.11.102.3" port: 34161 } }
I20260620 06:10:47.055472 30640 heartbeater.cc:503] Master 127.11.102.62:33531 was elected leader, sending a full tablet report...
I20260620 06:10:47.055794 30788 ts_tablet_manager.cc:1428] T 8a3245cbce6b4de392e5645a1a28f5e5 P 00991649baf741c18408069f4e7cbc76: Time spent starting tablet: real 0.005s user 0.000s sys 0.004s
I20260620 06:10:47.055799 30789 ts_tablet_manager.cc:1428] T 8a3245cbce6b4de392e5645a1a28f5e5 P 16ee41aaa52c43f581f1e5018f99ee95: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260620 06:10:47.056094 30771 heartbeater.cc:503] Master 127.11.102.62:33531 was elected leader, sending a full tablet report...
I20260620 06:10:47.056213 30790 tablet_bootstrap.cc:492] T 8a3245cbce6b4de392e5645a1a28f5e5 P cfd4b9d8635344bda1af0be90e9fa821: No bootstrap required, opened a new log
I20260620 06:10:47.056391 30790 ts_tablet_manager.cc:1397] T 8a3245cbce6b4de392e5645a1a28f5e5 P cfd4b9d8635344bda1af0be90e9fa821: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.000s
I20260620 06:10:47.060357 30790 raft_consensus.cc:348] T 8a3245cbce6b4de392e5645a1a28f5e5 P cfd4b9d8635344bda1af0be90e9fa821 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfd4b9d8635344bda1af0be90e9fa821" member_type: VOTER last_known_addr { host: "127.11.102.1" port: 46053 } } peers { permanent_uuid: "16ee41aaa52c43f581f1e5018f99ee95" member_type: VOTER last_known_addr { host: "127.11.102.2" port: 40143 } } peers { permanent_uuid: "00991649baf741c18408069f4e7cbc76" member_type: VOTER last_known_addr { host: "127.11.102.3" port: 34161 } }
I20260620 06:10:47.060631 30790 raft_consensus.cc:374] T 8a3245cbce6b4de392e5645a1a28f5e5 P cfd4b9d8635344bda1af0be90e9fa821 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 06:10:47.060730 30790 raft_consensus.cc:729] T 8a3245cbce6b4de392e5645a1a28f5e5 P cfd4b9d8635344bda1af0be90e9fa821 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cfd4b9d8635344bda1af0be90e9fa821, State: Initialized, Role: FOLLOWER
I20260620 06:10:47.061174 30790 consensus_queue.cc:260] T 8a3245cbce6b4de392e5645a1a28f5e5 P cfd4b9d8635344bda1af0be90e9fa821 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfd4b9d8635344bda1af0be90e9fa821" member_type: VOTER last_known_addr { host: "127.11.102.1" port: 46053 } } peers { permanent_uuid: "16ee41aaa52c43f581f1e5018f99ee95" member_type: VOTER last_known_addr { host: "127.11.102.2" port: 40143 } } peers { permanent_uuid: "00991649baf741c18408069f4e7cbc76" member_type: VOTER last_known_addr { host: "127.11.102.3" port: 34161 } }
I20260620 06:10:47.061749 30509 heartbeater.cc:503] Master 127.11.102.62:33531 was elected leader, sending a full tablet report...
I20260620 06:10:47.062012 30790 ts_tablet_manager.cc:1428] T 8a3245cbce6b4de392e5645a1a28f5e5 P cfd4b9d8635344bda1af0be90e9fa821: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
W20260620 06:10:47.087630 30510 tablet.cc:2369] T 8a3245cbce6b4de392e5645a1a28f5e5 P cfd4b9d8635344bda1af0be90e9fa821: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 06:10:47.245169 30772 tablet.cc:2369] T 8a3245cbce6b4de392e5645a1a28f5e5 P 00991649baf741c18408069f4e7cbc76: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 06:10:47.301390 30641 tablet.cc:2369] T 8a3245cbce6b4de392e5645a1a28f5e5 P 16ee41aaa52c43f581f1e5018f99ee95: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 06:10:47.359745 30796 raft_consensus.cc:482] T 8a3245cbce6b4de392e5645a1a28f5e5 P cfd4b9d8635344bda1af0be90e9fa821 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 06:10:47.359928 30796 raft_consensus.cc:504] T 8a3245cbce6b4de392e5645a1a28f5e5 P cfd4b9d8635344bda1af0be90e9fa821 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfd4b9d8635344bda1af0be90e9fa821" member_type: VOTER last_known_addr { host: "127.11.102.1" port: 46053 } } peers { permanent_uuid: "16ee41aaa52c43f581f1e5018f99ee95" member_type: VOTER last_known_addr { host: "127.11.102.2" port: 40143 } } peers { permanent_uuid: "00991649baf741c18408069f4e7cbc76" member_type: VOTER last_known_addr { host: "127.11.102.3" port: 34161 } }
I20260620 06:10:47.361085 30796 leader_election.cc:288] T 8a3245cbce6b4de392e5645a1a28f5e5 P cfd4b9d8635344bda1af0be90e9fa821 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 16ee41aaa52c43f581f1e5018f99ee95 (127.11.102.2:40143), 00991649baf741c18408069f4e7cbc76 (127.11.102.3:34161)
I20260620 06:10:47.366151 30787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:10:47.361864 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.102.1:51033 (local address 127.11.102.3:34161)
0620 06:10:47.361977 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0620 06:10:47.361982 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 06:10:47.362198 (+ 216us) server_negotiation.cc:407] Connection header received
0620 06:10:47.362276 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:10:47.362278 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:10:47.362340 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:10:47.362419 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 06:10:47.362664 (+ 245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:10:47.363374 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:10:47.364755 (+ 1381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:10:47.365373 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:10:47.365502 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:10:47.365553 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0620 06:10:47.365938 (+ 385us) server_negotiation.cc:299] Negotiation successful
0620 06:10:47.366011 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260620 06:10:47.366151 30786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:10:47.361479 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.102.1:39253 (local address 127.11.102.2:40143)
0620 06:10:47.361674 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0620 06:10:47.361680 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 06:10:47.361745 (+ 65us) server_negotiation.cc:407] Connection header received
0620 06:10:47.361916 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:10:47.361920 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:10:47.361975 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:10:47.362064 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 06:10:47.362368 (+ 304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:10:47.363165 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:10:47.364755 (+ 1590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:10:47.365373 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:10:47.365502 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:10:47.365559 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0620 06:10:47.365938 (+ 379us) server_negotiation.cc:299] Negotiation successful
0620 06:10:47.366022 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260620 06:10:47.366233 30797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:10:47.361362 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.102.2:40143 (local address 127.11.102.1:39253)
0620 06:10:47.361618 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0620 06:10:47.361632 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 06:10:47.361788 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:10:47.362079 (+ 291us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:10:47.362082 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:10:47.362101 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 06:10:47.362252 (+ 151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:10:47.362260 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:10:47.363269 (+ 1009us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:10:47.363271 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:10:47.364615 (+ 1344us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:10:47.364627 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:10:47.365505 (+ 878us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:10:47.365508 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:10:47.365790 (+ 282us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:10:47.365833 (+ 43us) client_negotiation.cc:769] Sending connection context
0620 06:10:47.365911 (+ 78us) client_negotiation.cc:240] Negotiation successful
0620 06:10:47.366050 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":175,"mutex_wait_us":139,"thread_start_us":72,"threads_started":1}
I20260620 06:10:47.366232 30798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:10:47.361742 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.102.3:34161 (local address 127.11.102.1:51033)
0620 06:10:47.362095 (+ 353us) negotiation.cc:107] Waiting for socket to connect
0620 06:10:47.362107 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 06:10:47.362194 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:10:47.362446 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:10:47.362448 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:10:47.362465 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 06:10:47.362556 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:10:47.362560 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:10:47.363468 (+ 908us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:10:47.363471 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:10:47.364639 (+ 1168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:10:47.364645 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:10:47.365504 (+ 859us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:10:47.365508 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:10:47.365785 (+ 277us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:10:47.365830 (+ 45us) client_negotiation.cc:769] Sending connection context
0620 06:10:47.365911 (+ 81us) client_negotiation.cc:240] Negotiation successful
0620 06:10:47.366050 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":292,"mutex_wait_us":106,"thread_start_us":132,"threads_started":1}
I20260620 06:10:47.366926 30725 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a3245cbce6b4de392e5645a1a28f5e5" candidate_uuid: "cfd4b9d8635344bda1af0be90e9fa821" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00991649baf741c18408069f4e7cbc76" is_pre_election: true
I20260620 06:10:47.366918 30594 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a3245cbce6b4de392e5645a1a28f5e5" candidate_uuid: "cfd4b9d8635344bda1af0be90e9fa821" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "16ee41aaa52c43f581f1e5018f99ee95" is_pre_election: true
I20260620 06:10:47.367230 30594 raft_consensus.cc:2454] T 8a3245cbce6b4de392e5645a1a28f5e5 P 16ee41aaa52c43f581f1e5018f99ee95 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cfd4b9d8635344bda1af0be90e9fa821 in term 0.
I20260620 06:10:47.367230 30725 raft_consensus.cc:2454] T 8a3245cbce6b4de392e5645a1a28f5e5 P 00991649baf741c18408069f4e7cbc76 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cfd4b9d8635344bda1af0be90e9fa821 in term 0.
I20260620 06:10:47.367774 30396 leader_election.cc:302] T 8a3245cbce6b4de392e5645a1a28f5e5 P cfd4b9d8635344bda1af0be90e9fa821 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 16ee41aaa52c43f581f1e5018f99ee95, cfd4b9d8635344bda1af0be90e9fa821; no voters:
I20260620 06:10:47.368009 30796 raft_consensus.cc:2781] T 8a3245cbce6b4de392e5645a1a28f5e5 P cfd4b9d8635344bda1af0be90e9fa821 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 06:10:47.368096 30796 raft_consensus.cc:482] T 8a3245cbce6b4de392e5645a1a28f5e5 P cfd4b9d8635344bda1af0be90e9fa821 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 06:10:47.368140 30796 raft_consensus.cc:3037] T 8a3245cbce6b4de392e5645a1a28f5e5 P cfd4b9d8635344bda1af0be90e9fa821 [term 0 FOLLOWER]: Advancing to term 1
I20260620 06:10:47.369251 30796 raft_consensus.cc:504] T 8a3245cbce6b4de392e5645a1a28f5e5 P cfd4b9d8635344bda1af0be90e9fa821 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfd4b9d8635344bda1af0be90e9fa821" member_type: VOTER last_known_addr { host: "127.11.102.1" port: 46053 } } peers { permanent_uuid: "16ee41aaa52c43f581f1e5018f99ee95" member_type: VOTER last_known_addr { host: "127.11.102.2" port: 40143 } } peers { permanent_uuid: "00991649baf741c18408069f4e7cbc76" member_type: VOTER last_known_addr { host: "127.11.102.3" port: 34161 } }
I20260620 06:10:47.369585 30796 leader_election.cc:288] T 8a3245cbce6b4de392e5645a1a28f5e5 P cfd4b9d8635344bda1af0be90e9fa821 [CANDIDATE]: Term 1 election: Requested vote from peers 16ee41aaa52c43f581f1e5018f99ee95 (127.11.102.2:40143), 00991649baf741c18408069f4e7cbc76 (127.11.102.3:34161)
I20260620 06:10:47.369921 30594 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a3245cbce6b4de392e5645a1a28f5e5" candidate_uuid: "cfd4b9d8635344bda1af0be90e9fa821" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "16ee41aaa52c43f581f1e5018f99ee95"
I20260620 06:10:47.369921 30725 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a3245cbce6b4de392e5645a1a28f5e5" candidate_uuid: "cfd4b9d8635344bda1af0be90e9fa821" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00991649baf741c18408069f4e7cbc76"
I20260620 06:10:47.370061 30594 raft_consensus.cc:3037] T 8a3245cbce6b4de392e5645a1a28f5e5 P 16ee41aaa52c43f581f1e5018f99ee95 [term 0 FOLLOWER]: Advancing to term 1
I20260620 06:10:47.370050 30725 raft_consensus.cc:3037] T 8a3245cbce6b4de392e5645a1a28f5e5 P 00991649baf741c18408069f4e7cbc76 [term 0 FOLLOWER]: Advancing to term 1
I20260620 06:10:47.371307 30594 raft_consensus.cc:2454] T 8a3245cbce6b4de392e5645a1a28f5e5 P 16ee41aaa52c43f581f1e5018f99ee95 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cfd4b9d8635344bda1af0be90e9fa821 in term 1.
I20260620 06:10:47.371307 30725 raft_consensus.cc:2454] T 8a3245cbce6b4de392e5645a1a28f5e5 P 00991649baf741c18408069f4e7cbc76 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cfd4b9d8635344bda1af0be90e9fa821 in term 1.
I20260620 06:10:47.371594 30396 leader_election.cc:302] T 8a3245cbce6b4de392e5645a1a28f5e5 P cfd4b9d8635344bda1af0be90e9fa821 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 16ee41aaa52c43f581f1e5018f99ee95, cfd4b9d8635344bda1af0be90e9fa821; no voters:
I20260620 06:10:47.371814 30796 raft_consensus.cc:2781] T 8a3245cbce6b4de392e5645a1a28f5e5 P cfd4b9d8635344bda1af0be90e9fa821 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 06:10:47.372058 30796 raft_consensus.cc:686] T 8a3245cbce6b4de392e5645a1a28f5e5 P cfd4b9d8635344bda1af0be90e9fa821 [term 1 LEADER]: Becoming Leader. State: Replica: cfd4b9d8635344bda1af0be90e9fa821, State: Running, Role: LEADER
I20260620 06:10:47.372424 30796 consensus_queue.cc:237] T 8a3245cbce6b4de392e5645a1a28f5e5 P cfd4b9d8635344bda1af0be90e9fa821 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfd4b9d8635344bda1af0be90e9fa821" member_type: VOTER last_known_addr { host: "127.11.102.1" port: 46053 } } peers { permanent_uuid: "16ee41aaa52c43f581f1e5018f99ee95" member_type: VOTER last_known_addr { host: "127.11.102.2" port: 40143 } } peers { permanent_uuid: "00991649baf741c18408069f4e7cbc76" member_type: VOTER last_known_addr { host: "127.11.102.3" port: 34161 } }
I20260620 06:10:47.375509 30319 catalog_manager.cc:5495] T 8a3245cbce6b4de392e5645a1a28f5e5 P cfd4b9d8635344bda1af0be90e9fa821 reported cstate change: term changed from 0 to 1, leader changed from <none> to cfd4b9d8635344bda1af0be90e9fa821 (127.11.102.1). New cstate: current_term: 1 leader_uuid: "cfd4b9d8635344bda1af0be90e9fa821" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfd4b9d8635344bda1af0be90e9fa821" member_type: VOTER last_known_addr { host: "127.11.102.1" port: 46053 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "16ee41aaa52c43f581f1e5018f99ee95" member_type: VOTER last_known_addr { host: "127.11.102.2" port: 40143 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "00991649baf741c18408069f4e7cbc76" member_type: VOTER last_known_addr { host: "127.11.102.3" port: 34161 } health_report { overall_health: UNKNOWN } } }
I20260620 06:10:47.397508 30785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:10:47.394053 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39708 (local address 127.11.102.1:46053)
0620 06:10:47.394181 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0620 06:10:47.394185 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 06:10:47.394198 (+ 13us) server_negotiation.cc:407] Connection header received
0620 06:10:47.394344 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:10:47.394346 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:10:47.394402 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:10:47.394494 (+ 92us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 06:10:47.394707 (+ 213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:10:47.395206 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:10:47.396096 (+ 890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:10:47.396313 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:10:47.396402 (+ 89us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:10:47.396698 (+ 296us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 06:10:47.397106 (+ 408us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 06:10:47.397214 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0620 06:10:47.397293 (+ 79us) server_negotiation.cc:299] Negotiation successful
0620 06:10:47.397324 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260620 06:10:47.403446 30725 raft_consensus.cc:1261] T 8a3245cbce6b4de392e5645a1a28f5e5 P 00991649baf741c18408069f4e7cbc76 [term 1 FOLLOWER]: Refusing update from remote peer cfd4b9d8635344bda1af0be90e9fa821: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 06:10:47.403417 30594 raft_consensus.cc:1261] T 8a3245cbce6b4de392e5645a1a28f5e5 P 16ee41aaa52c43f581f1e5018f99ee95 [term 1 FOLLOWER]: Refusing update from remote peer cfd4b9d8635344bda1af0be90e9fa821: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 06:10:47.404183 30796 consensus_queue.cc:1035] T 8a3245cbce6b4de392e5645a1a28f5e5 P cfd4b9d8635344bda1af0be90e9fa821 [LEADER]: Connected to new peer: Peer: permanent_uuid: "00991649baf741c18408069f4e7cbc76" member_type: VOTER last_known_addr { host: "127.11.102.3" port: 34161 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 06:10:47.404372 30799 consensus_queue.cc:1035] T 8a3245cbce6b4de392e5645a1a28f5e5 P cfd4b9d8635344bda1af0be90e9fa821 [LEADER]: Connected to new peer: Peer: permanent_uuid: "16ee41aaa52c43f581f1e5018f99ee95" member_type: VOTER last_known_addr { host: "127.11.102.2" port: 40143 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 06:10:47.412353 30804 mvcc.cc:205] Tried to move back new op lower bound from 7298809230956556288 to 7298809230840168448. Current Snapshot: MvccSnapshot[applied={T|T < 7298809230956556288}]
I20260620 06:10:47.420966 30319 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:55710:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 06:10:47.421159 30319 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:55710:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 06:10:47.423583 30319 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 48630a39279645f8a4eec0a6c561ee67: Sending DeleteTablet for 3 replicas of tablet 8a3245cbce6b4de392e5645a1a28f5e5
I20260620 06:10:47.424364 30443 tablet_service.cc:1514] Processing DeleteTablet for tablet 8a3245cbce6b4de392e5645a1a28f5e5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 06:10:47 UTC) from {username='slave'} at 127.0.0.1:39694
I20260620 06:10:47.424399 30574 tablet_service.cc:1514] Processing DeleteTablet for tablet 8a3245cbce6b4de392e5645a1a28f5e5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 06:10:47 UTC) from {username='slave'} at 127.0.0.1:51518
I20260620 06:10:47.424451 30705 tablet_service.cc:1514] Processing DeleteTablet for tablet 8a3245cbce6b4de392e5645a1a28f5e5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 06:10:47 UTC) from {username='slave'} at 127.0.0.1:51382
I20260620 06:10:47.424889 30813 tablet_replica.cc:330] T 8a3245cbce6b4de392e5645a1a28f5e5 P cfd4b9d8635344bda1af0be90e9fa821: stopping tablet replica
I20260620 06:10:47.425055 30812 tablet_replica.cc:330] T 8a3245cbce6b4de392e5645a1a28f5e5 P 16ee41aaa52c43f581f1e5018f99ee95: stopping tablet replica
I20260620 06:10:47.425276 30814 tablet_replica.cc:330] T 8a3245cbce6b4de392e5645a1a28f5e5 P 00991649baf741c18408069f4e7cbc76: stopping tablet replica
I20260620 06:10:47.425272 30813 raft_consensus.cc:2229] T 8a3245cbce6b4de392e5645a1a28f5e5 P cfd4b9d8635344bda1af0be90e9fa821 [term 1 LEADER]: Raft consensus shutting down.
I20260620 06:10:47.425522 30812 raft_consensus.cc:2229] T 8a3245cbce6b4de392e5645a1a28f5e5 P 16ee41aaa52c43f581f1e5018f99ee95 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 06:10:47.425550 30814 raft_consensus.cc:2229] T 8a3245cbce6b4de392e5645a1a28f5e5 P 00991649baf741c18408069f4e7cbc76 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 06:10:47.425626 11672 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9HMp43/build/debug/bin/kudu with pid 30381
I20260620 06:10:47.425660 30813 raft_consensus.cc:2258] T 8a3245cbce6b4de392e5645a1a28f5e5 P cfd4b9d8635344bda1af0be90e9fa821 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 06:10:47.425818 30814 raft_consensus.cc:2258] T 8a3245cbce6b4de392e5645a1a28f5e5 P 00991649baf741c18408069f4e7cbc76 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 06:10:47.427245 30814 ts_tablet_manager.cc:1905] T 8a3245cbce6b4de392e5645a1a28f5e5 P 00991649baf741c18408069f4e7cbc76: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 06:10:47.427541 30812 raft_consensus.cc:2258] T 8a3245cbce6b4de392e5645a1a28f5e5 P 16ee41aaa52c43f581f1e5018f99ee95 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 06:10:47.428964 30812 ts_tablet_manager.cc:1905] T 8a3245cbce6b4de392e5645a1a28f5e5 P 16ee41aaa52c43f581f1e5018f99ee95: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 06:10:47.430371 30814 ts_tablet_manager.cc:1918] T 8a3245cbce6b4de392e5645a1a28f5e5 P 00991649baf741c18408069f4e7cbc76: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 06:10:47.430471 30814 log.cc:1196] T 8a3245cbce6b4de392e5645a1a28f5e5 P 00991649baf741c18408069f4e7cbc76: Deleting WAL directory at /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781935603120653-11672-0/minicluster-data/ts-2/wal/wals/8a3245cbce6b4de392e5645a1a28f5e5
I20260620 06:10:47.430783 30814 ts_tablet_manager.cc:1939] T 8a3245cbce6b4de392e5645a1a28f5e5 P 00991649baf741c18408069f4e7cbc76: Deleting consensus metadata
I20260620 06:10:47.431592 30812 ts_tablet_manager.cc:1918] T 8a3245cbce6b4de392e5645a1a28f5e5 P 16ee41aaa52c43f581f1e5018f99ee95: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 06:10:47.431720 30812 log.cc:1196] T 8a3245cbce6b4de392e5645a1a28f5e5 P 16ee41aaa52c43f581f1e5018f99ee95: Deleting WAL directory at /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781935603120653-11672-0/minicluster-data/ts-1/wal/wals/8a3245cbce6b4de392e5645a1a28f5e5
I20260620 06:10:47.432025 30812 ts_tablet_manager.cc:1939] T 8a3245cbce6b4de392e5645a1a28f5e5 P 16ee41aaa52c43f581f1e5018f99ee95: Deleting consensus metadata
I20260620 06:10:47.432088 30305 catalog_manager.cc:4841] TS 00991649baf741c18408069f4e7cbc76 (127.11.102.3:34161): tablet 8a3245cbce6b4de392e5645a1a28f5e5 (table test-table [id=f25b5f4c6ff644fd982c6e65227ffdfe]) successfully deleted
W20260620 06:10:47.432952 30308 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.11.102.1:46053 (error 108)
I20260620 06:10:47.433027 30305 catalog_manager.cc:4841] TS 16ee41aaa52c43f581f1e5018f99ee95 (127.11.102.2:40143): tablet 8a3245cbce6b4de392e5645a1a28f5e5 (table test-table [id=f25b5f4c6ff644fd982c6e65227ffdfe]) successfully deleted
I20260620 06:10:47.433678 11672 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9HMp43/build/debug/bin/kudu with pid 30512
I20260620 06:10:47.433972 30784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:10:47.433636 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.102.1:46053 (local address 127.0.0.1:39714)
0620 06:10:47.433789 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0620 06:10:47.433877 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.102.1:46053: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260620 06:10:47.434476 30308 catalog_manager.cc:4584] TS cfd4b9d8635344bda1af0be90e9fa821 (127.11.102.1:46053): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 8a3245cbce6b4de392e5645a1a28f5e5: Network error: Client connection negotiation failed: client connection to 127.11.102.1:46053: connect: Connection refused (error 111)
I20260620 06:10:47.439556 11672 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9HMp43/build/debug/bin/kudu with pid 30643
I20260620 06:10:47.444906 11672 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9HMp43/build/debug/bin/kudu with pid 30290
2026-06-20T06:10:47Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (1292 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781935603120653-11672-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 06:10:47 dist-test-slave-9hm5 krb5kdc[30818](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 06:10:47 dist-test-slave-9hm5 krb5kdc[30818](info): set up 2 sockets
Jun 20 06:10:47 dist-test-slave-9hm5 krb5kdc[30818](info): commencing operation
krb5kdc: starting...
W20260620 06:10:48.479856 11672 mini_kdc.cc:121] Time spent starting KDC: real 1.014s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781935603120653-11672-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781935603120653-11672-0/krb5kdc/test-user.keytab.
Jun 20 06:10:48 dist-test-slave-9hm5 krb5kdc[30818](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935848, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T06:10:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T06:10:48Z Disabled control of system clock
WARNING: no policy specified for kudu/127.11.102.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.102.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.102.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781935603120653-11672-0/krb5kdc/kudu_127.11.102.62.keytab.
Entry for principal kudu/127.11.102.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781935603120653-11672-0/krb5kdc/kudu_127.11.102.62.keytab.
I20260620 06:10:48.599272 11672 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9HMp43/build/debug/bin/kudu
/tmp/dist-test-task9HMp43/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781935603120653-11672-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781935603120653-11672-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781935603120653-11672-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781935603120653-11672-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.102.62:45723
--webserver_interface=127.11.102.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781935603120653-11672-0/krb5kdc/kudu_127.11.102.62.keytab
--principal=kudu/127.11.102.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.102.20:37449
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.102.62:45723
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781935603120653-11672-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781935603120653-11672-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781935603120653-11672-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 06:10:48.708433 30831 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:10:48.708709 30831 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:10:48.708750 30831 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:10:48.712731 30831 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 06:10:48.712795 30831 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:10:48.712814 30831 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:10:48.712833 30831 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 06:10:48.712848 30831 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 06:10:48.718111 30831 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.102.20:37449
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781935603120653-11672-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781935603120653-11672-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.102.62:45723
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781935603120653-11672-0/krb5kdc/kudu_127.11.102.62.keytab
--principal=kudu/127.11.102.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.102.62:45723
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781935603120653-11672-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.102.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30831
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781935603120653-11672-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 05:28:48 UTC on ccycloud.quasar-fixhms.root.comops.site
build id 4457020
I20260620 06:10:48.719120 30831 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:10:48.720227 30831 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 06:10:48.728389 30837 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:10:48.728382 30839 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:10:48.728397 30836 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:10:48.728734 30831 server_base.cc:1034] running on GCE node
I20260620 06:10:48.729266 30831 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:10:48.730270 30831 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:10:48.731452 30831 hybrid_clock.cc:648] HybridClock initialized: now 1781935848731433 us; error 30 us; skew 500 ppm
Jun 20 06:10:48 dist-test-slave-9hm5 krb5kdc[30818](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935848, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.102.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 06:10:48.734068 30831 init.cc:377] Logged in from keytab as kudu/127.11.102.62@KRBTEST.COM (short username kudu)
I20260620 06:10:48.734845 30831 webserver.cc:460] Webserver started at http://127.11.102.62:42083/ using document root <none> and password file <none>
I20260620 06:10:48.735330 30831 fs_manager.cc:359] Metadata directory not provided
I20260620 06:10:48.735385 30831 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:10:48.735548 30831 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 06:10:48.737519 30831 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781935603120653-11672-0/minicluster-data/master-0/data/instance:
uuid: "8e41184171774fa095142b7c7ca27f34"
format_stamp: "Formatted at 2026-06-20 06:10:48 on dist-test-slave-9hm5"
server_key: "fdc69fe0bf05efcdcc57523c1faf77d3"
server_key_iv: "c8d3ee14cab8b1f6610a21ec8378c31c"
server_key_version: "encryptionkey@0"
I20260620 06:10:48.738021 30831 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781935603120653-11672-0/minicluster-data/master-0/wal/instance:
uuid: "8e41184171774fa095142b7c7ca27f34"
format_stamp: "Formatted at 2026-06-20 06:10:48 on dist-test-slave-9hm5"
server_key: "fdc69fe0bf05efcdcc57523c1faf77d3"
server_key_iv: "c8d3ee14cab8b1f6610a21ec8378c31c"
server_key_version: "encryptionkey@0"
I20260620 06:10:48.741506 30831 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260620 06:10:48.743819 30846 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:10:48.744997 30831 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 06:10:48.745121 30831 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781935603120653-11672-0/minicluster-data/master-0/data,/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781935603120653-11672-0/minicluster-data/master-0/wal
uuid: "8e41184171774fa095142b7c7ca27f34"
format_stamp: "Formatted at 2026-06-20 06:10:48 on dist-test-slave-9hm5"
server_key: "fdc69fe0bf05efcdcc57523c1faf77d3"
server_key_iv: "c8d3ee14cab8b1f6610a21ec8378c31c"
server_key_version: "encryptionkey@0"
I20260620 06:10:48.745221 30831 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781935603120653-11672-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781935603120653-11672-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781935603120653-11672-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:10:48.756235 30831 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:10:48.758920 30831 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:10:48.759125 30831 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:10:48.766127 30831 rpc_server.cc:307] RPC server started. Bound to: 127.11.102.62:45723
I20260620 06:10:48.766155 30898 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.102.62:45723 every 8 connection(s)
I20260620 06:10:48.767055 30831 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781935603120653-11672-0/minicluster-data/master-0/data/info.pb
I20260620 06:10:48.769672 30899 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 06:10:48.774960 30899 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8e41184171774fa095142b7c7ca27f34: Bootstrap starting.
I20260620 06:10:48.775573 11672 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9HMp43/build/debug/bin/kudu as pid 30831
I20260620 06:10:48.775729 11672 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781935603120653-11672-0/minicluster-data/master-0/wal/instance
I20260620 06:10:48.775985 11672 external_mini_cluster.cc:1428] Setting key d7ecb5ca952fc5e7e67d781635855df9
I20260620 06:10:48.777714 30899 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8e41184171774fa095142b7c7ca27f34: Neither blocks nor log segments found. Creating new log.
I20260620 06:10:48.778573 30899 log.cc:824] T 00000000000000000000000000000000 P 8e41184171774fa095142b7c7ca27f34: Log is configured to *not* fsync() on all Append() calls
I20260620 06:10:48.780779 30899 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8e41184171774fa095142b7c7ca27f34: No bootstrap required, opened a new log
I20260620 06:10:48.784749 30899 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8e41184171774fa095142b7c7ca27f34 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e41184171774fa095142b7c7ca27f34" member_type: VOTER last_known_addr { host: "127.11.102.62" port: 45723 } }
I20260620 06:10:48.784991 30899 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8e41184171774fa095142b7c7ca27f34 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 06:10:48.785102 30899 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8e41184171774fa095142b7c7ca27f34 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8e41184171774fa095142b7c7ca27f34, State: Initialized, Role: FOLLOWER
Jun 20 06:10:48 dist-test-slave-9hm5 krb5kdc[30818](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935848, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.102.62@KRBTEST.COM
I20260620 06:10:48.785638 30899 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8e41184171774fa095142b7c7ca27f34 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e41184171774fa095142b7c7ca27f34" member_type: VOTER last_known_addr { host: "127.11.102.62" port: 45723 } }
I20260620 06:10:48.785805 30899 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8e41184171774fa095142b7c7ca27f34 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 06:10:48.785887 30899 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8e41184171774fa095142b7c7ca27f34 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 06:10:48.785987 30899 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8e41184171774fa095142b7c7ca27f34 [term 0 FOLLOWER]: Advancing to term 1
I20260620 06:10:48.787138 30899 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8e41184171774fa095142b7c7ca27f34 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e41184171774fa095142b7c7ca27f34" member_type: VOTER last_known_addr { host: "127.11.102.62" port: 45723 } }
I20260620 06:10:48.787477 30899 leader_election.cc:302] T 00000000000000000000000000000000 P 8e41184171774fa095142b7c7ca27f34 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8e41184171774fa095142b7c7ca27f34; no voters:
I20260620 06:10:48.787849 30899 leader_election.cc:288] T 00000000000000000000000000000000 P 8e41184171774fa095142b7c7ca27f34 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 06:10:48.787919 30904 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8e41184171774fa095142b7c7ca27f34 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 06:10:48.788120 30904 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8e41184171774fa095142b7c7ca27f34 [term 1 LEADER]: Becoming Leader. State: Replica: 8e41184171774fa095142b7c7ca27f34, State: Running, Role: LEADER
I20260620 06:10:48.788456 30904 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8e41184171774fa095142b7c7ca27f34 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e41184171774fa095142b7c7ca27f34" member_type: VOTER last_known_addr { host: "127.11.102.62" port: 45723 } }
I20260620 06:10:48.789096 30899 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8e41184171774fa095142b7c7ca27f34 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 06:10:48.789587 30904 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8e41184171774fa095142b7c7ca27f34 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8e41184171774fa095142b7c7ca27f34. Latest consensus state: current_term: 1 leader_uuid: "8e41184171774fa095142b7c7ca27f34" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e41184171774fa095142b7c7ca27f34" member_type: VOTER last_known_addr { host: "127.11.102.62" port: 45723 } } }
I20260620 06:10:48.789791 30904 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8e41184171774fa095142b7c7ca27f34 [sys.catalog]: This master's current role is: LEADER
I20260620 06:10:48.790004 30905 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8e41184171774fa095142b7c7ca27f34 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8e41184171774fa095142b7c7ca27f34" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e41184171774fa095142b7c7ca27f34" member_type: VOTER last_known_addr { host: "127.11.102.62" port: 45723 } } }
I20260620 06:10:48.790186 30905 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8e41184171774fa095142b7c7ca27f34 [sys.catalog]: This master's current role is: LEADER
I20260620 06:10:48.790373 30913 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 06:10:48.791401 30902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:10:48.777281 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54354 (local address 127.11.102.62:45723)
0620 06:10:48.777709 (+ 428us) server_negotiation.cc:206] Beginning negotiation
0620 06:10:48.777730 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0620 06:10:48.777773 (+ 43us) server_negotiation.cc:407] Connection header received
0620 06:10:48.778624 (+ 851us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:10:48.778647 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:10:48.779059 (+ 412us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:10:48.779402 (+ 343us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:10:48.780118 (+ 716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:10:48.781544 (+ 1426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:10:48.782390 (+ 846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:10:48.783079 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:10:48.783270 (+ 191us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:10:48.785925 (+ 2655us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:10:48.785934 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:10:48.785953 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:10:48.785986 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:10:48.787869 (+ 1883us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:10:48.788421 (+ 552us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:10:48.788428 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:10:48.788433 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:10:48.788560 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:10:48.788850 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:10:48.788853 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:10:48.788854 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:10:48.789275 (+ 421us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:10:48.789469 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0620 06:10:48.790126 (+ 657us) server_negotiation.cc:299] Negotiation successful
0620 06:10:48.790441 (+ 315us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":273,"thread_start_us":107,"threads_started":1}
I20260620 06:10:48.793883 30913 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 06:10:48.799649 30913 catalog_manager.cc:1347] Generated new cluster ID: 956eb7283961445184e9c633ba74a734
I20260620 06:10:48.799765 30913 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 06:10:48.808485 30913 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 06:10:48.809322 30913 catalog_manager.cc:1504] Loading token signing keys...
I20260620 06:10:48.817885 30913 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8e41184171774fa095142b7c7ca27f34: Generated new TSK 0
I20260620 06:10:48.818670 30913 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.11.102.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.102.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.102.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781935603120653-11672-0/krb5kdc/kudu_127.11.102.1.keytab.
Entry for principal kudu/127.11.102.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781935603120653-11672-0/krb5kdc/kudu_127.11.102.1.keytab.
I20260620 06:10:48.850421 11672 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9HMp43/build/debug/bin/kudu
/tmp/dist-test-task9HMp43/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781935603120653-11672-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781935603120653-11672-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781935603120653-11672-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781935603120653-11672-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.102.1:0
--local_ip_for_outbound_sockets=127.11.102.1
--webserver_interface=127.11.102.1
--webserver_port=0
--tserver_master_addrs=127.11.102.62:45723
--keytab_file=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781935603120653-11672-0/krb5kdc/kudu_127.11.102.1.keytab
--principal=kudu/127.11.102.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.102.20:37449
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781935603120653-11672-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781935603120653-11672-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781935603120653-11672-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 06:10:48.962452 30925 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:10:48.962858 30925 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:10:48.962934 30925 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:10:48.966913 30925 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:10:48.966965 30925 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:10:48.967022 30925 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.102.1
I20260620 06:10:48.972126 30925 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.102.20:37449
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781935603120653-11672-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781935603120653-11672-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781935603120653-11672-0/krb5kdc/kudu_127.11.102.1.keytab
--principal=kudu/127.11.102.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.102.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781935603120653-11672-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.11.102.1
--webserver_port=0
--tserver_master_addrs=127.11.102.62:45723
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30925
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.102.1
--log_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781935603120653-11672-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 05:28:48 UTC on ccycloud.quasar-fixhms.root.comops.site
build id 4457020
I20260620 06:10:48.973042 30925 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:10:48.973958 30925 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 06:10:48.983191 30930 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:10:48.983209 30933 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:10:48.983209 30931 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:10:48.983359 30925 server_base.cc:1034] running on GCE node
I20260620 06:10:48.983873 30925 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:10:48.984484 30925 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:10:48.985693 30925 hybrid_clock.cc:648] HybridClock initialized: now 1781935848985675 us; error 29 us; skew 500 ppm
Jun 20 06:10:48 dist-test-slave-9hm5 krb5kdc[30818](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935848, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.102.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 06:10:48.988456 30925 init.cc:377] Logged in from keytab as kudu/127.11.102.1@KRBTEST.COM (short username kudu)
I20260620 06:10:48.989336 30925 webserver.cc:460] Webserver started at http://127.11.102.1:45003/ using document root <none> and password file <none>
I20260620 06:10:48.989989 30925 fs_manager.cc:359] Metadata directory not provided
I20260620 06:10:48.990082 30925 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:10:48.990310 30925 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 06:10:48.992372 30925 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781935603120653-11672-0/minicluster-data/ts-0/data/instance:
uuid: "41d1c6f5342949c3af7ca3c57155d282"
format_stamp: "Formatted at 2026-06-20 06:10:48 on dist-test-slave-9hm5"
server_key: "0c868dedc6af4cc6542045b7e4951e6e"
server_key_iv: "24817140126fcb0babf997134de73d87"
server_key_version: "encryptionkey@0"
I20260620 06:10:48.992902 30925 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781935603120653-11672-0/minicluster-data/ts-0/wal/instance:
uuid: "41d1c6f5342949c3af7ca3c57155d282"
format_stamp: "Formatted at 2026-06-20 06:10:48 on dist-test-slave-9hm5"
server_key: "0c868dedc6af4cc6542045b7e4951e6e"
server_key_iv: "24817140126fcb0babf997134de73d87"
server_key_version: "encryptionkey@0"
I20260620 06:10:48.996454 30925 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 06:10:48.998822 30940 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:10:49.000100 30925 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 06:10:49.000236 30925 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781935603120653-11672-0/minicluster-data/ts-0/data,/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781935603120653-11672-0/minicluster-data/ts-0/wal
uuid: "41d1c6f5342949c3af7ca3c57155d282"
format_stamp: "Formatted at 2026-06-20 06:10:48 on dist-test-slave-9hm5"
server_key: "0c868dedc6af4cc6542045b7e4951e6e"
server_key_iv: "24817140126fcb0babf997134de73d87"
server_key_version: "encryptionkey@0"
I20260620 06:10:49.000342 30925 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781935603120653-11672-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781935603120653-11672-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781935603120653-11672-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:10:49.016126 30925 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:10:49.018816 30925 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:10:49.019012 30925 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:10:49.019582 30925 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 06:10:49.020398 30925 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 06:10:49.020467 30925 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 06:10:49.020531 30925 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 06:10:49.020581 30925 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 06:10:49.029023 30925 rpc_server.cc:307] RPC server started. Bound to: 127.11.102.1:33735
I20260620 06:10:49.029054 31053 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.102.1:33735 every 8 connection(s)
I20260620 06:10:49.029867 30925 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781935603120653-11672-0/minicluster-data/ts-0/data/info.pb
I20260620 06:10:49.037636 11672 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9HMp43/build/debug/bin/kudu as pid 30925
I20260620 06:10:49.037796 11672 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781935603120653-11672-0/minicluster-data/ts-0/wal/instance
I20260620 06:10:49.038169 11672 external_mini_cluster.cc:1428] Setting key 26aca7c7ec8566ec7e0a6f9dcebf3444
Jun 20 06:10:49 dist-test-slave-9hm5 krb5kdc[30818](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935848, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.102.1@KRBTEST.COM for kudu/127.11.102.62@KRBTEST.COM
I20260620 06:10:49.044113 30902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:10:49.031571 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.102.1:50819 (local address 127.11.102.62:45723)
0620 06:10:49.031762 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0620 06:10:49.031766 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 06:10:49.032566 (+ 800us) server_negotiation.cc:407] Connection header received
0620 06:10:49.033864 (+ 1298us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:10:49.033867 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:10:49.033926 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:10:49.034034 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:10:49.035180 (+ 1146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:10:49.035768 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:10:49.036898 (+ 1130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:10:49.037144 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:10:49.037257 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:10:49.040089 (+ 2832us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:10:49.040096 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:10:49.040101 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:10:49.040132 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:10:49.041858 (+ 1726us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:10:49.042507 (+ 649us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:10:49.042509 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:10:49.042510 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:10:49.042603 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:10:49.042995 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:10:49.042997 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:10:49.042998 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:10:49.043166 (+ 168us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:10:49.043238 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0620 06:10:49.043887 (+ 649us) server_negotiation.cc:299] Negotiation successful
0620 06:10:49.043992 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92}
I20260620 06:10:49.045360 31056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:10:49.032004 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.102.62:45723 (local address 127.11.102.1:50819)
0620 06:10:49.032417 (+ 413us) negotiation.cc:107] Waiting for socket to connect
0620 06:10:49.032458 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0620 06:10:49.033646 (+ 1188us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:10:49.034211 (+ 565us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:10:49.034218 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:10:49.034569 (+ 351us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:10:49.035030 (+ 461us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:10:49.035042 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:10:49.035890 (+ 848us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:10:49.035895 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:10:49.036771 (+ 876us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:10:49.036780 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:10:49.037294 (+ 514us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:10:49.037298 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:10:49.037463 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:10:49.038099 (+ 636us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:10:49.038121 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:10:49.039924 (+ 1803us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:10:49.041985 (+ 2061us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:10:49.041993 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:10:49.042010 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:10:49.042405 (+ 395us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:10:49.042695 (+ 290us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:10:49.042701 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:10:49.042703 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:10:49.042901 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:10:49.043270 (+ 369us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:10:49.043279 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:10:49.043605 (+ 326us) client_negotiation.cc:769] Sending connection context
0620 06:10:49.043855 (+ 250us) client_negotiation.cc:240] Negotiation successful
0620 06:10:49.044246 (+ 391us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":241,"thread_start_us":94,"threads_started":1}
I20260620 06:10:49.046681 31054 heartbeater.cc:347] Connected to a master server at 127.11.102.62:45723
I20260620 06:10:49.046927 31054 heartbeater.cc:464] Registering TS with master...
I20260620 06:10:49.047410 31054 heartbeater.cc:511] Master 127.11.102.62:45723 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.11.102.2@KRBTEST.COM; defaulting to no policy
I20260620 06:10:49.049010 30863 ts_manager.cc:194] Registered new tserver with Master: 41d1c6f5342949c3af7ca3c57155d282 (127.11.102.1:33735)
I20260620 06:10:49.050233 30863 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.102.1@KRBTEST.COM'} at 127.11.102.1:50819
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.102.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.102.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781935603120653-11672-0/krb5kdc/kudu_127.11.102.2.keytab.
Entry for principal kudu/127.11.102.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781935603120653-11672-0/krb5kdc/kudu_127.11.102.2.keytab.
I20260620 06:10:49.064934 11672 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9HMp43/build/debug/bin/kudu
/tmp/dist-test-task9HMp43/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781935603120653-11672-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781935603120653-11672-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781935603120653-11672-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781935603120653-11672-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.102.2:0
--local_ip_for_outbound_sockets=127.11.102.2
--webserver_interface=127.11.102.2
--webserver_port=0
--tserver_master_addrs=127.11.102.62:45723
--keytab_file=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781935603120653-11672-0/krb5kdc/kudu_127.11.102.2.keytab
--principal=kudu/127.11.102.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.102.20:37449
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781935603120653-11672-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781935603120653-11672-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781935603120653-11672-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 06:10:49.173568 31059 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:10:49.173851 31059 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:10:49.173924 31059 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:10:49.178098 31059 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:10:49.178180 31059 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:10:49.178267 31059 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.102.2
I20260620 06:10:49.183681 31059 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.102.20:37449
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781935603120653-11672-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781935603120653-11672-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781935603120653-11672-0/krb5kdc/kudu_127.11.102.2.keytab
--principal=kudu/127.11.102.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.102.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781935603120653-11672-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.11.102.2
--webserver_port=0
--tserver_master_addrs=127.11.102.62:45723
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31059
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.102.2
--log_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781935603120653-11672-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 05:28:48 UTC on ccycloud.quasar-fixhms.root.comops.site
build id 4457020
I20260620 06:10:49.184679 31059 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:10:49.185668 31059 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 06:10:49.195518 31064 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:10:49.195530 31067 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:10:49.195530 31065 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:10:49.195807 31059 server_base.cc:1034] running on GCE node
I20260620 06:10:49.196319 31059 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:10:49.196969 31059 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:10:49.198159 31059 hybrid_clock.cc:648] HybridClock initialized: now 1781935849198148 us; error 33 us; skew 500 ppm
Jun 20 06:10:49 dist-test-slave-9hm5 krb5kdc[30818](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935849, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.102.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 06:10:49.201061 31059 init.cc:377] Logged in from keytab as kudu/127.11.102.2@KRBTEST.COM (short username kudu)
I20260620 06:10:49.201992 31059 webserver.cc:460] Webserver started at http://127.11.102.2:34335/ using document root <none> and password file <none>
I20260620 06:10:49.202555 31059 fs_manager.cc:359] Metadata directory not provided
I20260620 06:10:49.202648 31059 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:10:49.202889 31059 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 06:10:49.205049 31059 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781935603120653-11672-0/minicluster-data/ts-1/data/instance:
uuid: "d7ff25c6aacf4edea2cfaa78655a1ecd"
format_stamp: "Formatted at 2026-06-20 06:10:49 on dist-test-slave-9hm5"
server_key: "f821f24239e2f06a0856d1e639ab6c36"
server_key_iv: "061ea4f08cc588967e1a5d2ffdb6dc0e"
server_key_version: "encryptionkey@0"
I20260620 06:10:49.205791 31059 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781935603120653-11672-0/minicluster-data/ts-1/wal/instance:
uuid: "d7ff25c6aacf4edea2cfaa78655a1ecd"
format_stamp: "Formatted at 2026-06-20 06:10:49 on dist-test-slave-9hm5"
server_key: "f821f24239e2f06a0856d1e639ab6c36"
server_key_iv: "061ea4f08cc588967e1a5d2ffdb6dc0e"
server_key_version: "encryptionkey@0"
I20260620 06:10:49.209707 31059 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260620 06:10:49.212165 31074 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:10:49.213495 31059 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 06:10:49.213657 31059 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781935603120653-11672-0/minicluster-data/ts-1/data,/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781935603120653-11672-0/minicluster-data/ts-1/wal
uuid: "d7ff25c6aacf4edea2cfaa78655a1ecd"
format_stamp: "Formatted at 2026-06-20 06:10:49 on dist-test-slave-9hm5"
server_key: "f821f24239e2f06a0856d1e639ab6c36"
server_key_iv: "061ea4f08cc588967e1a5d2ffdb6dc0e"
server_key_version: "encryptionkey@0"
I20260620 06:10:49.213775 31059 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781935603120653-11672-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781935603120653-11672-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781935603120653-11672-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:10:49.228106 31059 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:10:49.231415 31059 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:10:49.231659 31059 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:10:49.232250 31059 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 06:10:49.233050 31059 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 06:10:49.233120 31059 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 06:10:49.233186 31059 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 06:10:49.233237 31059 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 06:10:49.242233 31059 rpc_server.cc:307] RPC server started. Bound to: 127.11.102.2:46547
I20260620 06:10:49.242255 31187 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.102.2:46547 every 8 connection(s)
I20260620 06:10:49.243098 31059 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781935603120653-11672-0/minicluster-data/ts-1/data/info.pb
I20260620 06:10:49.251160 11672 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9HMp43/build/debug/bin/kudu as pid 31059
I20260620 06:10:49.251297 11672 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781935603120653-11672-0/minicluster-data/ts-1/wal/instance
I20260620 06:10:49.251633 11672 external_mini_cluster.cc:1428] Setting key d20bd86813c8da40227cfbcc1381461c
Jun 20 06:10:49 dist-test-slave-9hm5 krb5kdc[30818](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935849, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.102.2@KRBTEST.COM for kudu/127.11.102.62@KRBTEST.COM
I20260620 06:10:49.256608 30902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:10:49.244847 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.102.2:48419 (local address 127.11.102.62:45723)
0620 06:10:49.245012 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0620 06:10:49.245017 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 06:10:49.245782 (+ 765us) server_negotiation.cc:407] Connection header received
0620 06:10:49.247076 (+ 1294us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:10:49.247079 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:10:49.247124 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:10:49.247198 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:10:49.248348 (+ 1150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:10:49.248871 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:10:49.249956 (+ 1085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:10:49.250178 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:10:49.250257 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:10:49.252967 (+ 2710us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:10:49.252971 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:10:49.252974 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:10:49.252999 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:10:49.254473 (+ 1474us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:10:49.255012 (+ 539us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:10:49.255016 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:10:49.255018 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:10:49.255102 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:10:49.255443 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:10:49.255448 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:10:49.255449 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:10:49.255711 (+ 262us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:10:49.255817 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0620 06:10:49.256306 (+ 489us) server_negotiation.cc:299] Negotiation successful
0620 06:10:49.256431 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260620 06:10:49.257419 31190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:10:49.245221 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.102.62:45723 (local address 127.11.102.2:48419)
0620 06:10:49.245635 (+ 414us) negotiation.cc:107] Waiting for socket to connect
0620 06:10:49.245676 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0620 06:10:49.246881 (+ 1205us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:10:49.247346 (+ 465us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:10:49.247353 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:10:49.247759 (+ 406us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:10:49.248228 (+ 469us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:10:49.248242 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:10:49.248972 (+ 730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:10:49.248975 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:10:49.249818 (+ 843us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:10:49.249826 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:10:49.250303 (+ 477us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:10:49.250306 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:10:49.250442 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:10:49.251080 (+ 638us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:10:49.251105 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:10:49.252802 (+ 1697us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:10:49.254621 (+ 1819us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:10:49.254626 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:10:49.254641 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:10:49.254899 (+ 258us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:10:49.255195 (+ 296us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:10:49.255199 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:10:49.255201 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:10:49.255346 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:10:49.255815 (+ 469us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:10:49.255822 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:10:49.256077 (+ 255us) client_negotiation.cc:769] Sending connection context
0620 06:10:49.256265 (+ 188us) client_negotiation.cc:240] Negotiation successful
0620 06:10:49.256584 (+ 319us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":234,"thread_start_us":98,"threads_started":1}
I20260620 06:10:49.258831 31188 heartbeater.cc:347] Connected to a master server at 127.11.102.62:45723
I20260620 06:10:49.259183 31188 heartbeater.cc:464] Registering TS with master...
I20260620 06:10:49.259737 31188 heartbeater.cc:511] Master 127.11.102.62:45723 requested a full tablet report, sending...
I20260620 06:10:49.260848 30863 ts_manager.cc:194] Registered new tserver with Master: d7ff25c6aacf4edea2cfaa78655a1ecd (127.11.102.2:46547)
WARNING: no policy specified for kudu/127.11.102.3@KRBTEST.COM; defaulting to no policy
I20260620 06:10:49.261582 30863 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.102.2@KRBTEST.COM'} at 127.11.102.2:48419
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.102.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.102.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781935603120653-11672-0/krb5kdc/kudu_127.11.102.3.keytab.
Entry for principal kudu/127.11.102.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781935603120653-11672-0/krb5kdc/kudu_127.11.102.3.keytab.
I20260620 06:10:49.278277 11672 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9HMp43/build/debug/bin/kudu
/tmp/dist-test-task9HMp43/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781935603120653-11672-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781935603120653-11672-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781935603120653-11672-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781935603120653-11672-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.102.3:0
--local_ip_for_outbound_sockets=127.11.102.3
--webserver_interface=127.11.102.3
--webserver_port=0
--tserver_master_addrs=127.11.102.62:45723
--keytab_file=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781935603120653-11672-0/krb5kdc/kudu_127.11.102.3.keytab
--principal=kudu/127.11.102.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.102.20:37449
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781935603120653-11672-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781935603120653-11672-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781935603120653-11672-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 06:10:49.386976 31193 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:10:49.387229 31193 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:10:49.387272 31193 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:10:49.391628 31193 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:10:49.391695 31193 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:10:49.391774 31193 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.102.3
I20260620 06:10:49.397105 31193 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.102.20:37449
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781935603120653-11672-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781935603120653-11672-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781935603120653-11672-0/krb5kdc/kudu_127.11.102.3.keytab
--principal=kudu/127.11.102.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.102.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781935603120653-11672-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.11.102.3
--webserver_port=0
--tserver_master_addrs=127.11.102.62:45723
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31193
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.102.3
--log_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781935603120653-11672-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 05:28:48 UTC on ccycloud.quasar-fixhms.root.comops.site
build id 4457020
I20260620 06:10:49.398038 31193 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:10:49.398967 31193 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 06:10:49.408453 31199 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:10:49.408447 31201 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:10:49.408473 31198 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:10:49.408701 31193 server_base.cc:1034] running on GCE node
I20260620 06:10:49.409229 31193 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:10:49.409857 31193 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:10:49.411031 31193 hybrid_clock.cc:648] HybridClock initialized: now 1781935849411020 us; error 31 us; skew 500 ppm
Jun 20 06:10:49 dist-test-slave-9hm5 krb5kdc[30818](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935849, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.102.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 06:10:49.413695 31193 init.cc:377] Logged in from keytab as kudu/127.11.102.3@KRBTEST.COM (short username kudu)
I20260620 06:10:49.414582 31193 webserver.cc:460] Webserver started at http://127.11.102.3:35425/ using document root <none> and password file <none>
I20260620 06:10:49.415144 31193 fs_manager.cc:359] Metadata directory not provided
I20260620 06:10:49.415233 31193 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:10:49.415457 31193 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 06:10:49.417642 31193 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781935603120653-11672-0/minicluster-data/ts-2/data/instance:
uuid: "20d4cfb9dded4b7a9755c576906d0662"
format_stamp: "Formatted at 2026-06-20 06:10:49 on dist-test-slave-9hm5"
server_key: "970c7423a4edfbb508c44e53591fde11"
server_key_iv: "07e2bb672496c3998b1bb0532e1f0cc5"
server_key_version: "encryptionkey@0"
I20260620 06:10:49.418205 31193 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781935603120653-11672-0/minicluster-data/ts-2/wal/instance:
uuid: "20d4cfb9dded4b7a9755c576906d0662"
format_stamp: "Formatted at 2026-06-20 06:10:49 on dist-test-slave-9hm5"
server_key: "970c7423a4edfbb508c44e53591fde11"
server_key_iv: "07e2bb672496c3998b1bb0532e1f0cc5"
server_key_version: "encryptionkey@0"
I20260620 06:10:49.421826 31193 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260620 06:10:49.424233 31208 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:10:49.425417 31193 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.003s
I20260620 06:10:49.425544 31193 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781935603120653-11672-0/minicluster-data/ts-2/data,/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781935603120653-11672-0/minicluster-data/ts-2/wal
uuid: "20d4cfb9dded4b7a9755c576906d0662"
format_stamp: "Formatted at 2026-06-20 06:10:49 on dist-test-slave-9hm5"
server_key: "970c7423a4edfbb508c44e53591fde11"
server_key_iv: "07e2bb672496c3998b1bb0532e1f0cc5"
server_key_version: "encryptionkey@0"
I20260620 06:10:49.425648 31193 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781935603120653-11672-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781935603120653-11672-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781935603120653-11672-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:10:49.450212 31193 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:10:49.453233 31193 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:10:49.453437 31193 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:10:49.454083 31193 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 06:10:49.454905 31193 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 06:10:49.454973 31193 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 06:10:49.455044 31193 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 06:10:49.455083 31193 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 06:10:49.464211 31193 rpc_server.cc:307] RPC server started. Bound to: 127.11.102.3:40039
I20260620 06:10:49.464251 31321 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.102.3:40039 every 8 connection(s)
I20260620 06:10:49.465132 31193 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781935603120653-11672-0/minicluster-data/ts-2/data/info.pb
I20260620 06:10:49.474260 11672 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9HMp43/build/debug/bin/kudu as pid 31193
I20260620 06:10:49.474359 11672 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781935603120653-11672-0/minicluster-data/ts-2/wal/instance
Jun 20 06:10:49 dist-test-slave-9hm5 krb5kdc[30818](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935849, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.102.3@KRBTEST.COM for kudu/127.11.102.62@KRBTEST.COM
I20260620 06:10:49.474679 11672 external_mini_cluster.cc:1428] Setting key bd265e098ec7d19f22ee64797335f43b
I20260620 06:10:49.478543 30902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:10:49.466990 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.102.3:49417 (local address 127.11.102.62:45723)
0620 06:10:49.467137 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0620 06:10:49.467140 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 06:10:49.468027 (+ 887us) server_negotiation.cc:407] Connection header received
0620 06:10:49.469298 (+ 1271us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:10:49.469301 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:10:49.469360 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:10:49.469438 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:10:49.470606 (+ 1168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:10:49.471146 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:10:49.472315 (+ 1169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:10:49.472550 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:10:49.472671 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:10:49.475197 (+ 2526us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:10:49.475201 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:10:49.475204 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:10:49.475231 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:10:49.476853 (+ 1622us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:10:49.477310 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:10:49.477312 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:10:49.477313 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:10:49.477386 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:10:49.477671 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:10:49.477673 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:10:49.477674 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:10:49.477829 (+ 155us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:10:49.477902 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0620 06:10:49.478347 (+ 445us) server_negotiation.cc:299] Negotiation successful
0620 06:10:49.478432 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260620 06:10:49.479375 31324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:10:49.467391 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.102.62:45723 (local address 127.11.102.3:49417)
0620 06:10:49.467879 (+ 488us) negotiation.cc:107] Waiting for socket to connect
0620 06:10:49.467921 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0620 06:10:49.469079 (+ 1158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:10:49.469609 (+ 530us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:10:49.469617 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:10:49.469970 (+ 353us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:10:49.470451 (+ 481us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:10:49.470465 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:10:49.471274 (+ 809us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:10:49.471278 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:10:49.472182 (+ 904us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:10:49.472191 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:10:49.472709 (+ 518us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:10:49.472713 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:10:49.472876 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:10:49.473402 (+ 526us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:10:49.473417 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:10:49.475003 (+ 1586us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:10:49.476959 (+ 1956us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:10:49.476963 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:10:49.476974 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:10:49.477226 (+ 252us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:10:49.477462 (+ 236us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:10:49.477464 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:10:49.477466 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:10:49.477595 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:10:49.477914 (+ 319us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:10:49.477918 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:10:49.478150 (+ 232us) client_negotiation.cc:769] Sending connection context
0620 06:10:49.478303 (+ 153us) client_negotiation.cc:240] Negotiation successful
0620 06:10:49.478540 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":305,"thread_start_us":159,"threads_started":1}
I20260620 06:10:49.480513 31322 heartbeater.cc:347] Connected to a master server at 127.11.102.62:45723
I20260620 06:10:49.480751 31322 heartbeater.cc:464] Registering TS with master...
I20260620 06:10:49.481223 31322 heartbeater.cc:511] Master 127.11.102.62:45723 requested a full tablet report, sending...
I20260620 06:10:49.482172 30863 ts_manager.cc:194] Registered new tserver with Master: 20d4cfb9dded4b7a9755c576906d0662 (127.11.102.3:40039)
I20260620 06:10:49.482689 30863 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.11.102.3@KRBTEST.COM'} at 127.11.102.3:49417
I20260620 06:10:49.488394 11672 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 06:10:49.497166 30902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:10:49.490368 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54358 (local address 127.11.102.62:45723)
0620 06:10:49.490481 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0620 06:10:49.490484 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 06:10:49.490689 (+ 205us) server_negotiation.cc:407] Connection header received
0620 06:10:49.490823 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:10:49.490825 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:10:49.490861 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:10:49.490920 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:10:49.491508 (+ 588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:10:49.492137 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:10:49.492917 (+ 780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:10:49.493115 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:10:49.493187 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:10:49.494380 (+ 1193us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:10:49.494382 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:10:49.494384 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:10:49.494404 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:10:49.495688 (+ 1284us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:10:49.496048 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:10:49.496050 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:10:49.496051 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:10:49.496121 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:10:49.496380 (+ 259us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:10:49.496383 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:10:49.496384 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:10:49.496635 (+ 251us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:10:49.496758 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0620 06:10:49.496952 (+ 194us) server_negotiation.cc:299] Negotiation successful
0620 06:10:49.497051 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260620 06:10:49.499971 30863 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54358:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 06:10:49.502864 30863 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 06:10:49.517730 31335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:10:49.511967 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41644 (local address 127.11.102.3:40039)
0620 06:10:49.513242 (+ 1275us) server_negotiation.cc:206] Beginning negotiation
0620 06:10:49.513252 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 06:10:49.513270 (+ 18us) server_negotiation.cc:407] Connection header received
0620 06:10:49.513331 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:10:49.513335 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:10:49.513473 (+ 138us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:10:49.514013 (+ 540us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 06:10:49.514163 (+ 150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:10:49.514856 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:10:49.516316 (+ 1460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:10:49.517030 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:10:49.517143 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:10:49.517292 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0620 06:10:49.517474 (+ 182us) server_negotiation.cc:299] Negotiation successful
0620 06:10:49.517580 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1180,"thread_start_us":196,"threads_started":1}
I20260620 06:10:49.517786 31333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:10:49.511797 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.102.3:40039 (local address 127.0.0.1:41644)
0620 06:10:49.512612 (+ 815us) negotiation.cc:107] Waiting for socket to connect
0620 06:10:49.512664 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0620 06:10:49.512796 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:10:49.513637 (+ 841us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:10:49.513643 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:10:49.513688 (+ 45us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 06:10:49.513863 (+ 175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:10:49.513870 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:10:49.514985 (+ 1115us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:10:49.514989 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:10:49.516195 (+ 1206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:10:49.516204 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:10:49.517156 (+ 952us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:10:49.517159 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:10:49.517270 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:10:49.517313 (+ 43us) client_negotiation.cc:769] Sending connection context
0620 06:10:49.517445 (+ 132us) client_negotiation.cc:240] Negotiation successful
0620 06:10:49.517567 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":622,"mutex_wait_us":5,"spinlock_wait_cycles":6528,"thread_start_us":77,"threads_started":1}
I20260620 06:10:49.518648 31334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:10:49.512016 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56138 (local address 127.11.102.1:33735)
0620 06:10:49.513398 (+ 1382us) server_negotiation.cc:206] Beginning negotiation
0620 06:10:49.513414 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0620 06:10:49.513433 (+ 19us) server_negotiation.cc:407] Connection header received
0620 06:10:49.513500 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:10:49.513504 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:10:49.513715 (+ 211us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:10:49.513886 (+ 171us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 06:10:49.514091 (+ 205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:10:49.515262 (+ 1171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:10:49.516991 (+ 1729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:10:49.517827 (+ 836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:10:49.518171 (+ 344us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:10:49.518315 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0620 06:10:49.518379 (+ 64us) server_negotiation.cc:299] Negotiation successful
0620 06:10:49.518473 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1291,"thread_start_us":107,"threads_started":1}
I20260620 06:10:49.519023 31337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:10:49.512114 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.102.2:46547 (local address 127.0.0.1:40492)
0620 06:10:49.512611 (+ 497us) negotiation.cc:107] Waiting for socket to connect
0620 06:10:49.512664 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0620 06:10:49.512796 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:10:49.513637 (+ 841us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:10:49.513644 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:10:49.513704 (+ 60us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 06:10:49.513886 (+ 182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:10:49.513895 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:10:49.515433 (+ 1538us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:10:49.515436 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:10:49.516806 (+ 1370us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:10:49.516818 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:10:49.518381 (+ 1563us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:10:49.518387 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:10:49.518491 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:10:49.518515 (+ 24us) client_negotiation.cc:769] Sending connection context
0620 06:10:49.518596 (+ 81us) client_negotiation.cc:240] Negotiation successful
0620 06:10:49.518670 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":418,"mutex_wait_us":81,"spinlock_wait_cycles":72320,"thread_start_us":88,"threads_started":1}
I20260620 06:10:49.519208 31338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:10:49.512589 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40492 (local address 127.11.102.2:46547)
0620 06:10:49.513126 (+ 537us) server_negotiation.cc:206] Beginning negotiation
0620 06:10:49.513138 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 06:10:49.513164 (+ 26us) server_negotiation.cc:407] Connection header received
0620 06:10:49.513243 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:10:49.513250 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:10:49.513455 (+ 205us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:10:49.513623 (+ 168us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 06:10:49.514038 (+ 415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:10:49.515267 (+ 1229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:10:49.516967 (+ 1700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:10:49.518071 (+ 1104us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:10:49.518170 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:10:49.518315 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0620 06:10:49.518924 (+ 609us) server_negotiation.cc:299] Negotiation successful
0620 06:10:49.519035 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":413,"thread_start_us":103,"threads_started":1}
I20260620 06:10:49.519855 31336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:10:49.511797 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.102.1:33735 (local address 127.0.0.1:56138)
0620 06:10:49.513023 (+ 1226us) negotiation.cc:107] Waiting for socket to connect
0620 06:10:49.513039 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 06:10:49.513164 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:10:49.513865 (+ 701us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:10:49.513867 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:10:49.513884 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 06:10:49.513981 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:10:49.513986 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:10:49.515433 (+ 1447us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:10:49.515436 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:10:49.516826 (+ 1390us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:10:49.516833 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:10:49.518176 (+ 1343us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:10:49.518180 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:10:49.518267 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:10:49.518289 (+ 22us) client_negotiation.cc:769] Sending connection context
0620 06:10:49.518351 (+ 62us) client_negotiation.cc:240] Negotiation successful
0620 06:10:49.519720 (+ 1369us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1150,"mutex_wait_us":756,"thread_start_us":90,"threads_started":1}
I20260620 06:10:49.520789 31256 tablet_service.cc:1467] Processing CreateTablet for tablet 3d83439842d54ecfb3a551e7b2918402 (DEFAULT_TABLE table=test-table [id=dea7f48bf3ca4ac2bfc153659b7108c9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 06:10:49.521735 31256 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3d83439842d54ecfb3a551e7b2918402. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 06:10:49.522511 30988 tablet_service.cc:1467] Processing CreateTablet for tablet 3d83439842d54ecfb3a551e7b2918402 (DEFAULT_TABLE table=test-table [id=dea7f48bf3ca4ac2bfc153659b7108c9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 06:10:49.522500 31122 tablet_service.cc:1467] Processing CreateTablet for tablet 3d83439842d54ecfb3a551e7b2918402 (DEFAULT_TABLE table=test-table [id=dea7f48bf3ca4ac2bfc153659b7108c9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 06:10:49.523536 31122 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3d83439842d54ecfb3a551e7b2918402. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 06:10:49.523536 30988 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3d83439842d54ecfb3a551e7b2918402. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 06:10:49.528862 31339 tablet_bootstrap.cc:492] T 3d83439842d54ecfb3a551e7b2918402 P 20d4cfb9dded4b7a9755c576906d0662: Bootstrap starting.
I20260620 06:10:49.531126 31339 tablet_bootstrap.cc:654] T 3d83439842d54ecfb3a551e7b2918402 P 20d4cfb9dded4b7a9755c576906d0662: Neither blocks nor log segments found. Creating new log.
I20260620 06:10:49.531136 31341 tablet_bootstrap.cc:492] T 3d83439842d54ecfb3a551e7b2918402 P d7ff25c6aacf4edea2cfaa78655a1ecd: Bootstrap starting.
I20260620 06:10:49.531508 31340 tablet_bootstrap.cc:492] T 3d83439842d54ecfb3a551e7b2918402 P 41d1c6f5342949c3af7ca3c57155d282: Bootstrap starting.
I20260620 06:10:49.532533 31339 log.cc:824] T 3d83439842d54ecfb3a551e7b2918402 P 20d4cfb9dded4b7a9755c576906d0662: Log is configured to *not* fsync() on all Append() calls
I20260620 06:10:49.533416 31340 tablet_bootstrap.cc:654] T 3d83439842d54ecfb3a551e7b2918402 P 41d1c6f5342949c3af7ca3c57155d282: Neither blocks nor log segments found. Creating new log.
I20260620 06:10:49.533713 31341 tablet_bootstrap.cc:654] T 3d83439842d54ecfb3a551e7b2918402 P d7ff25c6aacf4edea2cfaa78655a1ecd: Neither blocks nor log segments found. Creating new log.
I20260620 06:10:49.534485 31341 log.cc:824] T 3d83439842d54ecfb3a551e7b2918402 P d7ff25c6aacf4edea2cfaa78655a1ecd: Log is configured to *not* fsync() on all Append() calls
I20260620 06:10:49.534619 31340 log.cc:824] T 3d83439842d54ecfb3a551e7b2918402 P 41d1c6f5342949c3af7ca3c57155d282: Log is configured to *not* fsync() on all Append() calls
I20260620 06:10:49.534741 31339 tablet_bootstrap.cc:492] T 3d83439842d54ecfb3a551e7b2918402 P 20d4cfb9dded4b7a9755c576906d0662: No bootstrap required, opened a new log
I20260620 06:10:49.534943 31339 ts_tablet_manager.cc:1397] T 3d83439842d54ecfb3a551e7b2918402 P 20d4cfb9dded4b7a9755c576906d0662: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.004s
I20260620 06:10:49.536532 31340 tablet_bootstrap.cc:492] T 3d83439842d54ecfb3a551e7b2918402 P 41d1c6f5342949c3af7ca3c57155d282: No bootstrap required, opened a new log
I20260620 06:10:49.536587 31341 tablet_bootstrap.cc:492] T 3d83439842d54ecfb3a551e7b2918402 P d7ff25c6aacf4edea2cfaa78655a1ecd: No bootstrap required, opened a new log
I20260620 06:10:49.536752 31340 ts_tablet_manager.cc:1397] T 3d83439842d54ecfb3a551e7b2918402 P 41d1c6f5342949c3af7ca3c57155d282: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260620 06:10:49.536755 31341 ts_tablet_manager.cc:1397] T 3d83439842d54ecfb3a551e7b2918402 P d7ff25c6aacf4edea2cfaa78655a1ecd: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260620 06:10:49.539064 31339 raft_consensus.cc:348] T 3d83439842d54ecfb3a551e7b2918402 P 20d4cfb9dded4b7a9755c576906d0662 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20d4cfb9dded4b7a9755c576906d0662" member_type: VOTER last_known_addr { host: "127.11.102.3" port: 40039 } } peers { permanent_uuid: "41d1c6f5342949c3af7ca3c57155d282" member_type: VOTER last_known_addr { host: "127.11.102.1" port: 33735 } } peers { permanent_uuid: "d7ff25c6aacf4edea2cfaa78655a1ecd" member_type: VOTER last_known_addr { host: "127.11.102.2" port: 46547 } }
I20260620 06:10:49.539381 31339 raft_consensus.cc:374] T 3d83439842d54ecfb3a551e7b2918402 P 20d4cfb9dded4b7a9755c576906d0662 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 06:10:49.539443 31339 raft_consensus.cc:729] T 3d83439842d54ecfb3a551e7b2918402 P 20d4cfb9dded4b7a9755c576906d0662 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 20d4cfb9dded4b7a9755c576906d0662, State: Initialized, Role: FOLLOWER
I20260620 06:10:49.539940 31339 consensus_queue.cc:260] T 3d83439842d54ecfb3a551e7b2918402 P 20d4cfb9dded4b7a9755c576906d0662 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20d4cfb9dded4b7a9755c576906d0662" member_type: VOTER last_known_addr { host: "127.11.102.3" port: 40039 } } peers { permanent_uuid: "41d1c6f5342949c3af7ca3c57155d282" member_type: VOTER last_known_addr { host: "127.11.102.1" port: 33735 } } peers { permanent_uuid: "d7ff25c6aacf4edea2cfaa78655a1ecd" member_type: VOTER last_known_addr { host: "127.11.102.2" port: 46547 } }
I20260620 06:10:49.540637 31322 heartbeater.cc:503] Master 127.11.102.62:45723 was elected leader, sending a full tablet report...
I20260620 06:10:49.540719 31341 raft_consensus.cc:348] T 3d83439842d54ecfb3a551e7b2918402 P d7ff25c6aacf4edea2cfaa78655a1ecd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20d4cfb9dded4b7a9755c576906d0662" member_type: VOTER last_known_addr { host: "127.11.102.3" port: 40039 } } peers { permanent_uuid: "41d1c6f5342949c3af7ca3c57155d282" member_type: VOTER last_known_addr { host: "127.11.102.1" port: 33735 } } peers { permanent_uuid: "d7ff25c6aacf4edea2cfaa78655a1ecd" member_type: VOTER last_known_addr { host: "127.11.102.2" port: 46547 } }
I20260620 06:10:49.540964 31341 raft_consensus.cc:374] T 3d83439842d54ecfb3a551e7b2918402 P d7ff25c6aacf4edea2cfaa78655a1ecd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 06:10:49.540998 31339 ts_tablet_manager.cc:1428] T 3d83439842d54ecfb3a551e7b2918402 P 20d4cfb9dded4b7a9755c576906d0662: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260620 06:10:49.541024 31341 raft_consensus.cc:729] T 3d83439842d54ecfb3a551e7b2918402 P d7ff25c6aacf4edea2cfaa78655a1ecd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d7ff25c6aacf4edea2cfaa78655a1ecd, State: Initialized, Role: FOLLOWER
I20260620 06:10:49.541145 31340 raft_consensus.cc:348] T 3d83439842d54ecfb3a551e7b2918402 P 41d1c6f5342949c3af7ca3c57155d282 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20d4cfb9dded4b7a9755c576906d0662" member_type: VOTER last_known_addr { host: "127.11.102.3" port: 40039 } } peers { permanent_uuid: "41d1c6f5342949c3af7ca3c57155d282" member_type: VOTER last_known_addr { host: "127.11.102.1" port: 33735 } } peers { permanent_uuid: "d7ff25c6aacf4edea2cfaa78655a1ecd" member_type: VOTER last_known_addr { host: "127.11.102.2" port: 46547 } }
I20260620 06:10:49.541360 31340 raft_consensus.cc:374] T 3d83439842d54ecfb3a551e7b2918402 P 41d1c6f5342949c3af7ca3c57155d282 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 06:10:49.541412 31340 raft_consensus.cc:729] T 3d83439842d54ecfb3a551e7b2918402 P 41d1c6f5342949c3af7ca3c57155d282 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 41d1c6f5342949c3af7ca3c57155d282, State: Initialized, Role: FOLLOWER
I20260620 06:10:49.541496 31341 consensus_queue.cc:260] T 3d83439842d54ecfb3a551e7b2918402 P d7ff25c6aacf4edea2cfaa78655a1ecd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20d4cfb9dded4b7a9755c576906d0662" member_type: VOTER last_known_addr { host: "127.11.102.3" port: 40039 } } peers { permanent_uuid: "41d1c6f5342949c3af7ca3c57155d282" member_type: VOTER last_known_addr { host: "127.11.102.1" port: 33735 } } peers { permanent_uuid: "d7ff25c6aacf4edea2cfaa78655a1ecd" member_type: VOTER last_known_addr { host: "127.11.102.2" port: 46547 } }
I20260620 06:10:49.541836 31340 consensus_queue.cc:260] T 3d83439842d54ecfb3a551e7b2918402 P 41d1c6f5342949c3af7ca3c57155d282 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20d4cfb9dded4b7a9755c576906d0662" member_type: VOTER last_known_addr { host: "127.11.102.3" port: 40039 } } peers { permanent_uuid: "41d1c6f5342949c3af7ca3c57155d282" member_type: VOTER last_known_addr { host: "127.11.102.1" port: 33735 } } peers { permanent_uuid: "d7ff25c6aacf4edea2cfaa78655a1ecd" member_type: VOTER last_known_addr { host: "127.11.102.2" port: 46547 } }
I20260620 06:10:49.542418 31341 ts_tablet_manager.cc:1428] T 3d83439842d54ecfb3a551e7b2918402 P d7ff25c6aacf4edea2cfaa78655a1ecd: Time spent starting tablet: real 0.006s user 0.003s sys 0.004s
I20260620 06:10:49.542691 31340 ts_tablet_manager.cc:1428] T 3d83439842d54ecfb3a551e7b2918402 P 41d1c6f5342949c3af7ca3c57155d282: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260620 06:10:49.542760 31188 heartbeater.cc:503] Master 127.11.102.62:45723 was elected leader, sending a full tablet report...
I20260620 06:10:49.543824 31054 heartbeater.cc:503] Master 127.11.102.62:45723 was elected leader, sending a full tablet report...
I20260620 06:10:49.690506 31345 raft_consensus.cc:482] T 3d83439842d54ecfb3a551e7b2918402 P 20d4cfb9dded4b7a9755c576906d0662 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 06:10:49.690725 31345 raft_consensus.cc:504] T 3d83439842d54ecfb3a551e7b2918402 P 20d4cfb9dded4b7a9755c576906d0662 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20d4cfb9dded4b7a9755c576906d0662" member_type: VOTER last_known_addr { host: "127.11.102.3" port: 40039 } } peers { permanent_uuid: "41d1c6f5342949c3af7ca3c57155d282" member_type: VOTER last_known_addr { host: "127.11.102.1" port: 33735 } } peers { permanent_uuid: "d7ff25c6aacf4edea2cfaa78655a1ecd" member_type: VOTER last_known_addr { host: "127.11.102.2" port: 46547 } }
I20260620 06:10:49.692008 31345 leader_election.cc:288] T 3d83439842d54ecfb3a551e7b2918402 P 20d4cfb9dded4b7a9755c576906d0662 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 41d1c6f5342949c3af7ca3c57155d282 (127.11.102.1:33735), d7ff25c6aacf4edea2cfaa78655a1ecd (127.11.102.2:46547)
I20260620 06:10:49.695720 31324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:10:49.692279 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.102.1:33735 (local address 127.11.102.3:42717)
0620 06:10:49.692398 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0620 06:10:49.692415 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 06:10:49.692510 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:10:49.692769 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:10:49.692772 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:10:49.692791 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 06:10:49.692909 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:10:49.692914 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:10:49.693844 (+ 930us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:10:49.693849 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:10:49.694626 (+ 777us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:10:49.694635 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:10:49.695264 (+ 629us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:10:49.695268 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:10:49.695379 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:10:49.695396 (+ 17us) client_negotiation.cc:769] Sending connection context
0620 06:10:49.695487 (+ 91us) client_negotiation.cc:240] Negotiation successful
0620 06:10:49.695568 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":42}
I20260620 06:10:49.695720 31334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:10:49.692376 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.102.3:42717 (local address 127.11.102.1:33735)
0620 06:10:49.692506 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0620 06:10:49.692511 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 06:10:49.692531 (+ 20us) server_negotiation.cc:407] Connection header received
0620 06:10:49.692597 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:10:49.692600 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:10:49.692642 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:10:49.692721 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 06:10:49.693048 (+ 327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:10:49.693737 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:10:49.694757 (+ 1020us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:10:49.695146 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:10:49.695272 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:10:49.695322 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0620 06:10:49.695502 (+ 180us) server_negotiation.cc:299] Negotiation successful
0620 06:10:49.695571 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260620 06:10:49.695968 31338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:10:49.692391 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.102.3:54135 (local address 127.11.102.2:46547)
0620 06:10:49.692518 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0620 06:10:49.692530 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 06:10:49.693060 (+ 530us) server_negotiation.cc:407] Connection header received
0620 06:10:49.693153 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:10:49.693155 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:10:49.693198 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:10:49.693256 (+ 58us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 06:10:49.693529 (+ 273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:10:49.693984 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:10:49.695005 (+ 1021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:10:49.695403 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:10:49.695536 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:10:49.695582 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0620 06:10:49.695795 (+ 213us) server_negotiation.cc:299] Negotiation successful
0620 06:10:49.695863 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260620 06:10:49.695982 31348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:10:49.692279 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.102.2:46547 (local address 127.11.102.3:54135)
0620 06:10:49.692936 (+ 657us) negotiation.cc:107] Waiting for socket to connect
0620 06:10:49.692948 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 06:10:49.693071 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:10:49.693288 (+ 217us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:10:49.693291 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:10:49.693309 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 06:10:49.693423 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:10:49.693429 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:10:49.694095 (+ 666us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:10:49.694099 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:10:49.694875 (+ 776us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:10:49.694882 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:10:49.695527 (+ 645us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:10:49.695530 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:10:49.695671 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:10:49.695689 (+ 18us) client_negotiation.cc:769] Sending connection context
0620 06:10:49.695777 (+ 88us) client_negotiation.cc:240] Negotiation successful
0620 06:10:49.695840 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":599,"mutex_wait_us":282,"thread_start_us":133,"threads_started":1}
I20260620 06:10:49.696468 31008 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d83439842d54ecfb3a551e7b2918402" candidate_uuid: "20d4cfb9dded4b7a9755c576906d0662" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "41d1c6f5342949c3af7ca3c57155d282" is_pre_election: true
I20260620 06:10:49.696554 31142 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d83439842d54ecfb3a551e7b2918402" candidate_uuid: "20d4cfb9dded4b7a9755c576906d0662" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d7ff25c6aacf4edea2cfaa78655a1ecd" is_pre_election: true
I20260620 06:10:49.696808 31008 raft_consensus.cc:2454] T 3d83439842d54ecfb3a551e7b2918402 P 41d1c6f5342949c3af7ca3c57155d282 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 20d4cfb9dded4b7a9755c576906d0662 in term 0.
I20260620 06:10:49.696808 31142 raft_consensus.cc:2454] T 3d83439842d54ecfb3a551e7b2918402 P d7ff25c6aacf4edea2cfaa78655a1ecd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 20d4cfb9dded4b7a9755c576906d0662 in term 0.
I20260620 06:10:49.697285 31210 leader_election.cc:302] T 3d83439842d54ecfb3a551e7b2918402 P 20d4cfb9dded4b7a9755c576906d0662 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 20d4cfb9dded4b7a9755c576906d0662, 41d1c6f5342949c3af7ca3c57155d282, d7ff25c6aacf4edea2cfaa78655a1ecd; no voters:
I20260620 06:10:49.697618 31345 raft_consensus.cc:2781] T 3d83439842d54ecfb3a551e7b2918402 P 20d4cfb9dded4b7a9755c576906d0662 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 06:10:49.697743 31345 raft_consensus.cc:482] T 3d83439842d54ecfb3a551e7b2918402 P 20d4cfb9dded4b7a9755c576906d0662 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 06:10:49.697824 31345 raft_consensus.cc:3037] T 3d83439842d54ecfb3a551e7b2918402 P 20d4cfb9dded4b7a9755c576906d0662 [term 0 FOLLOWER]: Advancing to term 1
I20260620 06:10:49.698849 31345 raft_consensus.cc:504] T 3d83439842d54ecfb3a551e7b2918402 P 20d4cfb9dded4b7a9755c576906d0662 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20d4cfb9dded4b7a9755c576906d0662" member_type: VOTER last_known_addr { host: "127.11.102.3" port: 40039 } } peers { permanent_uuid: "41d1c6f5342949c3af7ca3c57155d282" member_type: VOTER last_known_addr { host: "127.11.102.1" port: 33735 } } peers { permanent_uuid: "d7ff25c6aacf4edea2cfaa78655a1ecd" member_type: VOTER last_known_addr { host: "127.11.102.2" port: 46547 } }
I20260620 06:10:49.699182 31345 leader_election.cc:288] T 3d83439842d54ecfb3a551e7b2918402 P 20d4cfb9dded4b7a9755c576906d0662 [CANDIDATE]: Term 1 election: Requested vote from peers 41d1c6f5342949c3af7ca3c57155d282 (127.11.102.1:33735), d7ff25c6aacf4edea2cfaa78655a1ecd (127.11.102.2:46547)
I20260620 06:10:49.699539 31142 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d83439842d54ecfb3a551e7b2918402" candidate_uuid: "20d4cfb9dded4b7a9755c576906d0662" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d7ff25c6aacf4edea2cfaa78655a1ecd"
I20260620 06:10:49.699699 31008 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d83439842d54ecfb3a551e7b2918402" candidate_uuid: "20d4cfb9dded4b7a9755c576906d0662" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "41d1c6f5342949c3af7ca3c57155d282"
I20260620 06:10:49.699733 31142 raft_consensus.cc:3037] T 3d83439842d54ecfb3a551e7b2918402 P d7ff25c6aacf4edea2cfaa78655a1ecd [term 0 FOLLOWER]: Advancing to term 1
I20260620 06:10:49.699847 31008 raft_consensus.cc:3037] T 3d83439842d54ecfb3a551e7b2918402 P 41d1c6f5342949c3af7ca3c57155d282 [term 0 FOLLOWER]: Advancing to term 1
I20260620 06:10:49.700807 31142 raft_consensus.cc:2454] T 3d83439842d54ecfb3a551e7b2918402 P d7ff25c6aacf4edea2cfaa78655a1ecd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 20d4cfb9dded4b7a9755c576906d0662 in term 1.
I20260620 06:10:49.700817 31008 raft_consensus.cc:2454] T 3d83439842d54ecfb3a551e7b2918402 P 41d1c6f5342949c3af7ca3c57155d282 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 20d4cfb9dded4b7a9755c576906d0662 in term 1.
I20260620 06:10:49.701115 31211 leader_election.cc:302] T 3d83439842d54ecfb3a551e7b2918402 P 20d4cfb9dded4b7a9755c576906d0662 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20d4cfb9dded4b7a9755c576906d0662, d7ff25c6aacf4edea2cfaa78655a1ecd; no voters:
I20260620 06:10:49.701320 31345 raft_consensus.cc:2781] T 3d83439842d54ecfb3a551e7b2918402 P 20d4cfb9dded4b7a9755c576906d0662 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 06:10:49.701556 31345 raft_consensus.cc:686] T 3d83439842d54ecfb3a551e7b2918402 P 20d4cfb9dded4b7a9755c576906d0662 [term 1 LEADER]: Becoming Leader. State: Replica: 20d4cfb9dded4b7a9755c576906d0662, State: Running, Role: LEADER
I20260620 06:10:49.701871 31345 consensus_queue.cc:237] T 3d83439842d54ecfb3a551e7b2918402 P 20d4cfb9dded4b7a9755c576906d0662 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20d4cfb9dded4b7a9755c576906d0662" member_type: VOTER last_known_addr { host: "127.11.102.3" port: 40039 } } peers { permanent_uuid: "41d1c6f5342949c3af7ca3c57155d282" member_type: VOTER last_known_addr { host: "127.11.102.1" port: 33735 } } peers { permanent_uuid: "d7ff25c6aacf4edea2cfaa78655a1ecd" member_type: VOTER last_known_addr { host: "127.11.102.2" port: 46547 } }
I20260620 06:10:49.705112 30863 catalog_manager.cc:5495] T 3d83439842d54ecfb3a551e7b2918402 P 20d4cfb9dded4b7a9755c576906d0662 reported cstate change: term changed from 0 to 1, leader changed from <none> to 20d4cfb9dded4b7a9755c576906d0662 (127.11.102.3). New cstate: current_term: 1 leader_uuid: "20d4cfb9dded4b7a9755c576906d0662" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20d4cfb9dded4b7a9755c576906d0662" member_type: VOTER last_known_addr { host: "127.11.102.3" port: 40039 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "41d1c6f5342949c3af7ca3c57155d282" member_type: VOTER last_known_addr { host: "127.11.102.1" port: 33735 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d7ff25c6aacf4edea2cfaa78655a1ecd" member_type: VOTER last_known_addr { host: "127.11.102.2" port: 46547 } health_report { overall_health: UNKNOWN } } }
W20260620 06:10:49.716097 31323 tablet.cc:2369] T 3d83439842d54ecfb3a551e7b2918402 P 20d4cfb9dded4b7a9755c576906d0662: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 06:10:49.744186 31189 tablet.cc:2369] T 3d83439842d54ecfb3a551e7b2918402 P d7ff25c6aacf4edea2cfaa78655a1ecd: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 06:10:49.752307 31335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:10:49.747749 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41660 (local address 127.11.102.3:40039)
0620 06:10:49.747902 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0620 06:10:49.747909 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 06:10:49.747929 (+ 20us) server_negotiation.cc:407] Connection header received
0620 06:10:49.748000 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:10:49.748002 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:10:49.748068 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:10:49.748159 (+ 91us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 06:10:49.748428 (+ 269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:10:49.749022 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:10:49.750395 (+ 1373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:10:49.750763 (+ 368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:10:49.750872 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:10:49.751274 (+ 402us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 06:10:49.751854 (+ 580us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 06:10:49.751997 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0620 06:10:49.752048 (+ 51us) server_negotiation.cc:299] Negotiation successful
0620 06:10:49.752130 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260620 06:10:49.757783 31142 raft_consensus.cc:1261] T 3d83439842d54ecfb3a551e7b2918402 P d7ff25c6aacf4edea2cfaa78655a1ecd [term 1 FOLLOWER]: Refusing update from remote peer 20d4cfb9dded4b7a9755c576906d0662: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 06:10:49.757835 31008 raft_consensus.cc:1261] T 3d83439842d54ecfb3a551e7b2918402 P 41d1c6f5342949c3af7ca3c57155d282 [term 1 FOLLOWER]: Refusing update from remote peer 20d4cfb9dded4b7a9755c576906d0662: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 06:10:49.758463 31349 consensus_queue.cc:1035] T 3d83439842d54ecfb3a551e7b2918402 P 20d4cfb9dded4b7a9755c576906d0662 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7ff25c6aacf4edea2cfaa78655a1ecd" member_type: VOTER last_known_addr { host: "127.11.102.2" port: 46547 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 06:10:49.758638 31345 consensus_queue.cc:1035] T 3d83439842d54ecfb3a551e7b2918402 P 20d4cfb9dded4b7a9755c576906d0662 [LEADER]: Connected to new peer: Peer: permanent_uuid: "41d1c6f5342949c3af7ca3c57155d282" member_type: VOTER last_known_addr { host: "127.11.102.1" port: 33735 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 06:10:49.766167 31353 mvcc.cc:205] Tried to move back new op lower bound from 7298809240601264128 to 7298809240381587456. Current Snapshot: MvccSnapshot[applied={T|T < 7298809240601264128}]
I20260620 06:10:49.773962 30863 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54358:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 06:10:49.774139 30863 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54358:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 06:10:49.776463 30863 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8e41184171774fa095142b7c7ca27f34: Sending DeleteTablet for 3 replicas of tablet 3d83439842d54ecfb3a551e7b2918402
I20260620 06:10:49.777127 30988 tablet_service.cc:1514] Processing DeleteTablet for tablet 3d83439842d54ecfb3a551e7b2918402 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 06:10:49 UTC) from {username='kudu', principal='kudu/127.11.102.62@KRBTEST.COM'} at 127.0.0.1:56138
I20260620 06:10:49.777125 31256 tablet_service.cc:1514] Processing DeleteTablet for tablet 3d83439842d54ecfb3a551e7b2918402 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 06:10:49 UTC) from {username='kudu', principal='kudu/127.11.102.62@KRBTEST.COM'} at 127.0.0.1:41644
I20260620 06:10:49.777232 31122 tablet_service.cc:1514] Processing DeleteTablet for tablet 3d83439842d54ecfb3a551e7b2918402 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 06:10:49 UTC) from {username='kudu', principal='kudu/127.11.102.62@KRBTEST.COM'} at 127.0.0.1:40492
I20260620 06:10:49.777755 11672 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9HMp43/build/debug/bin/kudu with pid 30925
W20260620 06:10:49.787602 30848 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.11.102.1:33735 (error 108)
I20260620 06:10:49.787786 31362 tablet_replica.cc:330] T 3d83439842d54ecfb3a551e7b2918402 P 20d4cfb9dded4b7a9755c576906d0662: stopping tablet replica
I20260620 06:10:49.787839 11672 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9HMp43/build/debug/bin/kudu with pid 31059
I20260620 06:10:49.788033 31363 tablet_replica.cc:330] T 3d83439842d54ecfb3a551e7b2918402 P d7ff25c6aacf4edea2cfaa78655a1ecd: stopping tablet replica
I20260620 06:10:49.789273 31336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:10:49.788959 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.102.1:33735 (local address 127.0.0.1:56142)
0620 06:10:49.789136 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0620 06:10:49.789198 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.102.1:33735: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105}
W20260620 06:10:49.789438 30848 catalog_manager.cc:4584] TS 41d1c6f5342949c3af7ca3c57155d282 (127.11.102.1:33735): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 3d83439842d54ecfb3a551e7b2918402: Network error: Client connection negotiation failed: client connection to 127.11.102.1:33735: connect: Connection refused (error 111)
I20260620 06:10:49.790030 31362 raft_consensus.cc:2229] T 3d83439842d54ecfb3a551e7b2918402 P 20d4cfb9dded4b7a9755c576906d0662 [term 1 LEADER]: Raft consensus shutting down.
I20260620 06:10:49.790433 31362 raft_consensus.cc:2258] T 3d83439842d54ecfb3a551e7b2918402 P 20d4cfb9dded4b7a9755c576906d0662 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 06:10:49.791951 31362 ts_tablet_manager.cc:1905] T 3d83439842d54ecfb3a551e7b2918402 P 20d4cfb9dded4b7a9755c576906d0662: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 06:10:49.793826 11672 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9HMp43/build/debug/bin/kudu with pid 31193
I20260620 06:10:49.794601 31336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:10:49.793979 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.102.2:46547 (local address 127.0.0.1:40494)
0620 06:10:49.794468 (+ 489us) negotiation.cc:107] Waiting for socket to connect
0620 06:10:49.794532 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.102.2:46547: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":411}
I20260620 06:10:49.798933 11672 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9HMp43/build/debug/bin/kudu with pid 30831
2026-06-20T06:10:49Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (2355 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-06-20T06:10:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T06:10:49Z Disabled control of system clock
I20260620 06:10:49.828794 11672 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9HMp43/build/debug/bin/kudu
/tmp/dist-test-task9HMp43/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781935603120653-11672-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781935603120653-11672-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781935603120653-11672-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781935603120653-11672-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.102.62:36325
--webserver_interface=127.11.102.62
--webserver_port=0
--builtin_ntp_servers=127.11.102.20:41211
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.102.62:36325
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260620 06:10:49.937561 31368 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:10:49.937819 31368 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:10:49.937855 31368 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:10:49.941699 31368 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 06:10:49.941742 31368 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:10:49.941758 31368 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:10:49.941774 31368 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 06:10:49.941788 31368 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 06:10:49.946386 31368 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.102.20:41211
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781935603120653-11672-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781935603120653-11672-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.102.62:36325
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.102.62:36325
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781935603120653-11672-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.102.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31368
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781935603120653-11672-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 05:28:48 UTC on ccycloud.quasar-fixhms.root.comops.site
build id 4457020
I20260620 06:10:49.947358 31368 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:10:49.948688 31368 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 06:10:49.956949 31374 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:10:49.956933 31376 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:10:49.957037 31368 server_base.cc:1034] running on GCE node
W20260620 06:10:49.956933 31373 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:10:49.957736 31368 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:10:49.958743 31368 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:10:49.959928 31368 hybrid_clock.cc:648] HybridClock initialized: now 1781935849959891 us; error 50 us; skew 500 ppm
I20260620 06:10:49.961669 31368 webserver.cc:460] Webserver started at http://127.11.102.62:43883/ using document root <none> and password file <none>
I20260620 06:10:49.962234 31368 fs_manager.cc:359] Metadata directory not provided
I20260620 06:10:49.962327 31368 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:10:49.962555 31368 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 06:10:49.964633 31368 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781935603120653-11672-0/minicluster-data/master-0/data/instance:
uuid: "286eba4a81ba42e7b96d742071c0a075"
format_stamp: "Formatted at 2026-06-20 06:10:49 on dist-test-slave-9hm5"
server_key: "9c05c19688ea2e8b17bd6355e2d29213"
server_key_iv: "a4a98d52c1c161232d82a81f818e37f3"
server_key_version: "encryptionkey@0"
I20260620 06:10:49.965183 31368 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781935603120653-11672-0/minicluster-data/master-0/wal/instance:
uuid: "286eba4a81ba42e7b96d742071c0a075"
format_stamp: "Formatted at 2026-06-20 06:10:49 on dist-test-slave-9hm5"
server_key: "9c05c19688ea2e8b17bd6355e2d29213"
server_key_iv: "a4a98d52c1c161232d82a81f818e37f3"
server_key_version: "encryptionkey@0"
I20260620 06:10:49.968773 31368 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260620 06:10:49.971047 31382 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:10:49.972270 31368 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260620 06:10:49.972404 31368 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781935603120653-11672-0/minicluster-data/master-0/data,/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781935603120653-11672-0/minicluster-data/master-0/wal
uuid: "286eba4a81ba42e7b96d742071c0a075"
format_stamp: "Formatted at 2026-06-20 06:10:49 on dist-test-slave-9hm5"
server_key: "9c05c19688ea2e8b17bd6355e2d29213"
server_key_iv: "a4a98d52c1c161232d82a81f818e37f3"
server_key_version: "encryptionkey@0"
I20260620 06:10:49.972504 31368 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781935603120653-11672-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781935603120653-11672-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781935603120653-11672-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:10:49.986984 31368 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:10:49.987699 31368 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:10:49.987890 31368 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:10:49.994478 31368 rpc_server.cc:307] RPC server started. Bound to: 127.11.102.62:36325
I20260620 06:10:49.994493 31434 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.102.62:36325 every 8 connection(s)
I20260620 06:10:49.995344 31368 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781935603120653-11672-0/minicluster-data/master-0/data/info.pb
I20260620 06:10:49.998359 31435 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 06:10:50.003823 31435 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 286eba4a81ba42e7b96d742071c0a075: Bootstrap starting.
I20260620 06:10:50.004165 11672 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9HMp43/build/debug/bin/kudu as pid 31368
I20260620 06:10:50.004294 11672 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781935603120653-11672-0/minicluster-data/master-0/wal/instance
I20260620 06:10:50.004619 11672 external_mini_cluster.cc:1428] Setting key b62febbca2c004a13d97497fc8f8b839
I20260620 06:10:50.006574 31435 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 286eba4a81ba42e7b96d742071c0a075: Neither blocks nor log segments found. Creating new log.
I20260620 06:10:50.007285 31435 log.cc:824] T 00000000000000000000000000000000 P 286eba4a81ba42e7b96d742071c0a075: Log is configured to *not* fsync() on all Append() calls
I20260620 06:10:50.009130 31435 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 286eba4a81ba42e7b96d742071c0a075: No bootstrap required, opened a new log
I20260620 06:10:50.012415 31435 raft_consensus.cc:348] T 00000000000000000000000000000000 P 286eba4a81ba42e7b96d742071c0a075 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "286eba4a81ba42e7b96d742071c0a075" member_type: VOTER last_known_addr { host: "127.11.102.62" port: 36325 } }
I20260620 06:10:50.012624 31435 raft_consensus.cc:374] T 00000000000000000000000000000000 P 286eba4a81ba42e7b96d742071c0a075 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 06:10:50.012657 31435 raft_consensus.cc:729] T 00000000000000000000000000000000 P 286eba4a81ba42e7b96d742071c0a075 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 286eba4a81ba42e7b96d742071c0a075, State: Initialized, Role: FOLLOWER
I20260620 06:10:50.013200 31435 consensus_queue.cc:260] T 00000000000000000000000000000000 P 286eba4a81ba42e7b96d742071c0a075 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "286eba4a81ba42e7b96d742071c0a075" member_type: VOTER last_known_addr { host: "127.11.102.62" port: 36325 } }
I20260620 06:10:50.013381 31435 raft_consensus.cc:388] T 00000000000000000000000000000000 P 286eba4a81ba42e7b96d742071c0a075 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 06:10:50.013451 31435 raft_consensus.cc:482] T 00000000000000000000000000000000 P 286eba4a81ba42e7b96d742071c0a075 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 06:10:50.013553 31435 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 286eba4a81ba42e7b96d742071c0a075 [term 0 FOLLOWER]: Advancing to term 1
I20260620 06:10:50.014000 31438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:10:50.005986 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45760 (local address 127.11.102.62:36325)
0620 06:10:50.006654 (+ 668us) server_negotiation.cc:206] Beginning negotiation
0620 06:10:50.006676 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0620 06:10:50.006725 (+ 49us) server_negotiation.cc:407] Connection header received
0620 06:10:50.007379 (+ 654us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:10:50.007403 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:10:50.007912 (+ 509us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:10:50.008293 (+ 381us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:10:50.008706 (+ 413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:10:50.009710 (+ 1004us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:10:50.010586 (+ 876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:10:50.011042 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:10:50.011229 (+ 187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:10:50.012130 (+ 901us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:10:50.012136 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:10:50.012157 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 06:10:50.012519 (+ 362us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:10:50.012570 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 06:10:50.012589 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:10:50.012713 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0620 06:10:50.012929 (+ 216us) server_negotiation.cc:299] Negotiation successful
0620 06:10:50.013175 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":286,"thread_start_us":130,"threads_started":1}
I20260620 06:10:50.014482 31435 raft_consensus.cc:504] T 00000000000000000000000000000000 P 286eba4a81ba42e7b96d742071c0a075 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "286eba4a81ba42e7b96d742071c0a075" member_type: VOTER last_known_addr { host: "127.11.102.62" port: 36325 } }
I20260620 06:10:50.014819 31435 leader_election.cc:302] T 00000000000000000000000000000000 P 286eba4a81ba42e7b96d742071c0a075 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 286eba4a81ba42e7b96d742071c0a075; no voters:
I20260620 06:10:50.015138 31435 leader_election.cc:288] T 00000000000000000000000000000000 P 286eba4a81ba42e7b96d742071c0a075 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 06:10:50.015199 31440 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 286eba4a81ba42e7b96d742071c0a075 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 06:10:50.015421 31440 raft_consensus.cc:686] T 00000000000000000000000000000000 P 286eba4a81ba42e7b96d742071c0a075 [term 1 LEADER]: Becoming Leader. State: Replica: 286eba4a81ba42e7b96d742071c0a075, State: Running, Role: LEADER
I20260620 06:10:50.015725 31440 consensus_queue.cc:237] T 00000000000000000000000000000000 P 286eba4a81ba42e7b96d742071c0a075 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "286eba4a81ba42e7b96d742071c0a075" member_type: VOTER last_known_addr { host: "127.11.102.62" port: 36325 } }
I20260620 06:10:50.016376 31435 sys_catalog.cc:564] T 00000000000000000000000000000000 P 286eba4a81ba42e7b96d742071c0a075 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 06:10:50.016867 31440 sys_catalog.cc:455] T 00000000000000000000000000000000 P 286eba4a81ba42e7b96d742071c0a075 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "286eba4a81ba42e7b96d742071c0a075" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "286eba4a81ba42e7b96d742071c0a075" member_type: VOTER last_known_addr { host: "127.11.102.62" port: 36325 } } }
I20260620 06:10:50.017019 31440 sys_catalog.cc:458] T 00000000000000000000000000000000 P 286eba4a81ba42e7b96d742071c0a075 [sys.catalog]: This master's current role is: LEADER
I20260620 06:10:50.017279 31441 sys_catalog.cc:455] T 00000000000000000000000000000000 P 286eba4a81ba42e7b96d742071c0a075 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 286eba4a81ba42e7b96d742071c0a075. Latest consensus state: current_term: 1 leader_uuid: "286eba4a81ba42e7b96d742071c0a075" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "286eba4a81ba42e7b96d742071c0a075" member_type: VOTER last_known_addr { host: "127.11.102.62" port: 36325 } } }
I20260620 06:10:50.017407 31441 sys_catalog.cc:458] T 00000000000000000000000000000000 P 286eba4a81ba42e7b96d742071c0a075 [sys.catalog]: This master's current role is: LEADER
W20260620 06:10:50.021054 31455 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 286eba4a81ba42e7b96d742071c0a075: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 06:10:50.021150 31455 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 06:10:50.021234 31445 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 06:10:50.021893 31445 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 06:10:50.026930 31445 catalog_manager.cc:1347] Generated new cluster ID: fdd02b99c1b146bc905114a49dd1c4d1
I20260620 06:10:50.026989 31445 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 06:10:50.038089 31445 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 06:10:50.038938 31445 catalog_manager.cc:1504] Loading token signing keys...
I20260620 06:10:50.045290 31445 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 286eba4a81ba42e7b96d742071c0a075: Generated new TSK 0
I20260620 06:10:50.045912 31445 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 06:10:50.056730 11672 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9HMp43/build/debug/bin/kudu
/tmp/dist-test-task9HMp43/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781935603120653-11672-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781935603120653-11672-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781935603120653-11672-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781935603120653-11672-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.102.1:0
--local_ip_for_outbound_sockets=127.11.102.1
--webserver_interface=127.11.102.1
--webserver_port=0
--tserver_master_addrs=127.11.102.62:36325
--builtin_ntp_servers=127.11.102.20:41211
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260620 06:10:50.165923 31459 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:10:50.166222 31459 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:10:50.166289 31459 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:10:50.170245 31459 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:10:50.170293 31459 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:10:50.170344 31459 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.102.1
I20260620 06:10:50.174906 31459 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.102.20:41211
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781935603120653-11672-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781935603120653-11672-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.102.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781935603120653-11672-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.11.102.1
--webserver_port=0
--tserver_master_addrs=127.11.102.62:36325
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31459
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.102.1
--log_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781935603120653-11672-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 05:28:48 UTC on ccycloud.quasar-fixhms.root.comops.site
build id 4457020
I20260620 06:10:50.176081 31459 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:10:50.177472 31459 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 06:10:50.186928 31464 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:10:50.186918 31465 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:10:50.186904 31467 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:10:50.187275 31459 server_base.cc:1034] running on GCE node
I20260620 06:10:50.187763 31459 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:10:50.188371 31459 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:10:50.189545 31459 hybrid_clock.cc:648] HybridClock initialized: now 1781935850189512 us; error 46 us; skew 500 ppm
I20260620 06:10:50.191309 31459 webserver.cc:460] Webserver started at http://127.11.102.1:35877/ using document root <none> and password file <none>
I20260620 06:10:50.191937 31459 fs_manager.cc:359] Metadata directory not provided
I20260620 06:10:50.192000 31459 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:10:50.192209 31459 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 06:10:50.194175 31459 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781935603120653-11672-0/minicluster-data/ts-0/data/instance:
uuid: "0ba8283540404efd8c98de39a4a269ae"
format_stamp: "Formatted at 2026-06-20 06:10:50 on dist-test-slave-9hm5"
server_key: "70593fbb222021dd352dfe17abe37416"
server_key_iv: "9cb5fd3eb8fe4b3d2a8a94d18b328622"
server_key_version: "encryptionkey@0"
I20260620 06:10:50.194722 31459 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781935603120653-11672-0/minicluster-data/ts-0/wal/instance:
uuid: "0ba8283540404efd8c98de39a4a269ae"
format_stamp: "Formatted at 2026-06-20 06:10:50 on dist-test-slave-9hm5"
server_key: "70593fbb222021dd352dfe17abe37416"
server_key_iv: "9cb5fd3eb8fe4b3d2a8a94d18b328622"
server_key_version: "encryptionkey@0"
I20260620 06:10:50.198299 31459 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.006s
I20260620 06:10:50.200721 31473 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:10:50.201902 31459 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260620 06:10:50.202045 31459 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781935603120653-11672-0/minicluster-data/ts-0/data,/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781935603120653-11672-0/minicluster-data/ts-0/wal
uuid: "0ba8283540404efd8c98de39a4a269ae"
format_stamp: "Formatted at 2026-06-20 06:10:50 on dist-test-slave-9hm5"
server_key: "70593fbb222021dd352dfe17abe37416"
server_key_iv: "9cb5fd3eb8fe4b3d2a8a94d18b328622"
server_key_version: "encryptionkey@0"
I20260620 06:10:50.202154 31459 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781935603120653-11672-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781935603120653-11672-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781935603120653-11672-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:10:50.218806 31459 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:10:50.219449 31459 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:10:50.219710 31459 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:10:50.220274 31459 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 06:10:50.221169 31459 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 06:10:50.221239 31459 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 06:10:50.221305 31459 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 06:10:50.221354 31459 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 06:10:50.230162 31459 rpc_server.cc:307] RPC server started. Bound to: 127.11.102.1:44483
I20260620 06:10:50.230192 31586 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.102.1:44483 every 8 connection(s)
I20260620 06:10:50.231041 31459 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781935603120653-11672-0/minicluster-data/ts-0/data/info.pb
I20260620 06:10:50.233112 11672 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9HMp43/build/debug/bin/kudu as pid 31459
I20260620 06:10:50.233243 11672 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781935603120653-11672-0/minicluster-data/ts-0/wal/instance
I20260620 06:10:50.233560 11672 external_mini_cluster.cc:1428] Setting key 5a731591080a0bf71f07d43d81c95e3c
I20260620 06:10:50.235699 11672 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9HMp43/build/debug/bin/kudu
/tmp/dist-test-task9HMp43/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781935603120653-11672-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781935603120653-11672-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781935603120653-11672-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781935603120653-11672-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.102.2:0
--local_ip_for_outbound_sockets=127.11.102.2
--webserver_interface=127.11.102.2
--webserver_port=0
--tserver_master_addrs=127.11.102.62:36325
--builtin_ntp_servers=127.11.102.20:41211
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260620 06:10:50.241205 31438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:10:50.232961 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.102.1:47687 (local address 127.11.102.62:36325)
0620 06:10:50.233113 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0620 06:10:50.233117 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 06:10:50.234530 (+ 1413us) server_negotiation.cc:407] Connection header received
0620 06:10:50.236279 (+ 1749us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:10:50.236287 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:10:50.236348 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:10:50.236442 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:10:50.237599 (+ 1157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:10:50.238126 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:10:50.239287 (+ 1161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:10:50.239518 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:10:50.239666 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:10:50.240726 (+ 1060us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:10:50.240735 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:10:50.240737 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 06:10:50.240748 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:10:50.240766 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 06:10:50.240774 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:10:50.240852 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0620 06:10:50.241011 (+ 159us) server_negotiation.cc:299] Negotiation successful
0620 06:10:50.241068 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260620 06:10:50.242010 31589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:10:50.233529 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.102.62:36325 (local address 127.11.102.1:47687)
0620 06:10:50.234375 (+ 846us) negotiation.cc:107] Waiting for socket to connect
0620 06:10:50.234429 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0620 06:10:50.236066 (+ 1637us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:10:50.236601 (+ 535us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:10:50.236611 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:10:50.236694 (+ 83us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:10:50.237409 (+ 715us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:10:50.237428 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:10:50.238259 (+ 831us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:10:50.238263 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:10:50.239143 (+ 880us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:10:50.239152 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:10:50.239689 (+ 537us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:10:50.239694 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:10:50.239857 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:10:50.240481 (+ 624us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 06:10:50.240482 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:10:50.240520 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 06:10:50.240581 (+ 61us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:10:50.240855 (+ 274us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:10:50.240866 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:10:50.240873 (+ 7us) client_negotiation.cc:769] Sending connection context
0620 06:10:50.240998 (+ 125us) client_negotiation.cc:240] Negotiation successful
0620 06:10:50.241277 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":340,"thread_start_us":156,"threads_started":1}
I20260620 06:10:50.243394 31587 heartbeater.cc:347] Connected to a master server at 127.11.102.62:36325
I20260620 06:10:50.243693 31587 heartbeater.cc:464] Registering TS with master...
I20260620 06:10:50.244177 31587 heartbeater.cc:511] Master 127.11.102.62:36325 requested a full tablet report, sending...
I20260620 06:10:50.245836 31399 ts_manager.cc:194] Registered new tserver with Master: 0ba8283540404efd8c98de39a4a269ae (127.11.102.1:44483)
I20260620 06:10:50.247480 31399 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.11.102.1:47687
W20260620 06:10:50.345185 31590 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:10:50.345439 31590 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:10:50.345476 31590 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:10:50.349339 31590 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:10:50.349381 31590 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:10:50.349433 31590 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.102.2
I20260620 06:10:50.354065 31590 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.102.20:41211
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781935603120653-11672-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781935603120653-11672-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.102.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781935603120653-11672-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.11.102.2
--webserver_port=0
--tserver_master_addrs=127.11.102.62:36325
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31590
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.102.2
--log_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781935603120653-11672-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 05:28:48 UTC on ccycloud.quasar-fixhms.root.comops.site
build id 4457020
I20260620 06:10:50.355023 31590 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:10:50.356324 31590 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 06:10:50.365600 31595 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:10:50.365631 31598 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:10:50.365631 31596 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:10:50.365942 31590 server_base.cc:1034] running on GCE node
I20260620 06:10:50.366379 31590 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:10:50.367051 31590 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:10:50.368204 31590 hybrid_clock.cc:648] HybridClock initialized: now 1781935850368168 us; error 47 us; skew 500 ppm
I20260620 06:10:50.369897 31590 webserver.cc:460] Webserver started at http://127.11.102.2:35059/ using document root <none> and password file <none>
I20260620 06:10:50.370453 31590 fs_manager.cc:359] Metadata directory not provided
I20260620 06:10:50.370514 31590 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:10:50.370770 31590 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 06:10:50.372823 31590 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781935603120653-11672-0/minicluster-data/ts-1/data/instance:
uuid: "2202052527374fbab20b488c9fec4d7d"
format_stamp: "Formatted at 2026-06-20 06:10:50 on dist-test-slave-9hm5"
server_key: "80c2539d6848c0ce3b92a44b13e6b7f7"
server_key_iv: "8dcefe8123336ae6602d71163663a0ce"
server_key_version: "encryptionkey@0"
I20260620 06:10:50.373363 31590 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781935603120653-11672-0/minicluster-data/ts-1/wal/instance:
uuid: "2202052527374fbab20b488c9fec4d7d"
format_stamp: "Formatted at 2026-06-20 06:10:50 on dist-test-slave-9hm5"
server_key: "80c2539d6848c0ce3b92a44b13e6b7f7"
server_key_iv: "8dcefe8123336ae6602d71163663a0ce"
server_key_version: "encryptionkey@0"
I20260620 06:10:50.377070 31590 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 06:10:50.379395 31604 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:10:50.380600 31590 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 06:10:50.380791 31590 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781935603120653-11672-0/minicluster-data/ts-1/data,/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781935603120653-11672-0/minicluster-data/ts-1/wal
uuid: "2202052527374fbab20b488c9fec4d7d"
format_stamp: "Formatted at 2026-06-20 06:10:50 on dist-test-slave-9hm5"
server_key: "80c2539d6848c0ce3b92a44b13e6b7f7"
server_key_iv: "8dcefe8123336ae6602d71163663a0ce"
server_key_version: "encryptionkey@0"
I20260620 06:10:50.380900 31590 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781935603120653-11672-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781935603120653-11672-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781935603120653-11672-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:10:50.402508 31590 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:10:50.403173 31590 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:10:50.403366 31590 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:10:50.404019 31590 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 06:10:50.404989 31590 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 06:10:50.405055 31590 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 06:10:50.405122 31590 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 06:10:50.405169 31590 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 06:10:50.414012 31590 rpc_server.cc:307] RPC server started. Bound to: 127.11.102.2:33177
I20260620 06:10:50.414038 31717 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.102.2:33177 every 8 connection(s)
I20260620 06:10:50.414966 31590 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781935603120653-11672-0/minicluster-data/ts-1/data/info.pb
I20260620 06:10:50.422202 11672 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9HMp43/build/debug/bin/kudu as pid 31590
I20260620 06:10:50.422338 11672 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781935603120653-11672-0/minicluster-data/ts-1/wal/instance
I20260620 06:10:50.422713 11672 external_mini_cluster.cc:1428] Setting key aae879b74262eae411b88e6139cc9ddd
I20260620 06:10:50.424763 31438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:10:50.416863 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.102.2:53403 (local address 127.11.102.62:36325)
0620 06:10:50.417028 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0620 06:10:50.417034 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 06:10:50.418220 (+ 1186us) server_negotiation.cc:407] Connection header received
0620 06:10:50.419656 (+ 1436us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:10:50.419659 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:10:50.419719 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:10:50.419802 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:10:50.420728 (+ 926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:10:50.421269 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:10:50.422698 (+ 1429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:10:50.422975 (+ 277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:10:50.423065 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:10:50.424246 (+ 1181us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:10:50.424249 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:10:50.424252 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 06:10:50.424264 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:10:50.424286 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 06:10:50.424302 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:10:50.424391 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0620 06:10:50.424559 (+ 168us) server_negotiation.cc:299] Negotiation successful
0620 06:10:50.424621 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260620 06:10:50.425040 11672 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9HMp43/build/debug/bin/kudu
/tmp/dist-test-task9HMp43/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781935603120653-11672-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781935603120653-11672-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781935603120653-11672-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781935603120653-11672-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.102.3:0
--local_ip_for_outbound_sockets=127.11.102.3
--webserver_interface=127.11.102.3
--webserver_port=0
--tserver_master_addrs=127.11.102.62:36325
--builtin_ntp_servers=127.11.102.20:41211
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260620 06:10:50.425750 31720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:10:50.417310 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.102.62:36325 (local address 127.11.102.2:53403)
0620 06:10:50.418083 (+ 773us) negotiation.cc:107] Waiting for socket to connect
0620 06:10:50.418124 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0620 06:10:50.419419 (+ 1295us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:10:50.419972 (+ 553us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:10:50.419980 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:10:50.420072 (+ 92us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:10:50.420578 (+ 506us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:10:50.420596 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:10:50.421399 (+ 803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:10:50.421403 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:10:50.422558 (+ 1155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:10:50.422571 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:10:50.423144 (+ 573us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:10:50.423150 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:10:50.423306 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:10:50.423942 (+ 636us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 06:10:50.423944 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:10:50.423996 (+ 52us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 06:10:50.424094 (+ 98us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:10:50.424408 (+ 314us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:10:50.424415 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:10:50.424420 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 06:10:50.424532 (+ 112us) client_negotiation.cc:240] Negotiation successful
0620 06:10:50.424777 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":286,"thread_start_us":112,"threads_started":1}
I20260620 06:10:50.426931 31718 heartbeater.cc:347] Connected to a master server at 127.11.102.62:36325
I20260620 06:10:50.427176 31718 heartbeater.cc:464] Registering TS with master...
I20260620 06:10:50.427758 31718 heartbeater.cc:511] Master 127.11.102.62:36325 requested a full tablet report, sending...
I20260620 06:10:50.428843 31399 ts_manager.cc:194] Registered new tserver with Master: 2202052527374fbab20b488c9fec4d7d (127.11.102.2:33177)
I20260620 06:10:50.429451 31399 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.11.102.2:53403
W20260620 06:10:50.535362 31721 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:10:50.535655 31721 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:10:50.535737 31721 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:10:50.539656 31721 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:10:50.539729 31721 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:10:50.539825 31721 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.102.3
I20260620 06:10:50.544567 31721 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.102.20:41211
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781935603120653-11672-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781935603120653-11672-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.102.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781935603120653-11672-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.11.102.3
--webserver_port=0
--tserver_master_addrs=127.11.102.62:36325
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31721
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.102.3
--log_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781935603120653-11672-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 05:28:48 UTC on ccycloud.quasar-fixhms.root.comops.site
build id 4457020
I20260620 06:10:50.545594 31721 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:10:50.546990 31721 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 06:10:50.556445 31726 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:10:50.556476 31727 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:10:50.556476 31729 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:10:50.556731 31721 server_base.cc:1034] running on GCE node
I20260620 06:10:50.557155 31721 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:10:50.557763 31721 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:10:50.558931 31721 hybrid_clock.cc:648] HybridClock initialized: now 1781935850558910 us; error 30 us; skew 500 ppm
I20260620 06:10:50.560794 31721 webserver.cc:460] Webserver started at http://127.11.102.3:36703/ using document root <none> and password file <none>
I20260620 06:10:50.561362 31721 fs_manager.cc:359] Metadata directory not provided
I20260620 06:10:50.561426 31721 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:10:50.561745 31721 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 06:10:50.563935 31721 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781935603120653-11672-0/minicluster-data/ts-2/data/instance:
uuid: "80262711a64f406aaabbeb5458210619"
format_stamp: "Formatted at 2026-06-20 06:10:50 on dist-test-slave-9hm5"
server_key: "704eab533512bf9e29f13577b72aec22"
server_key_iv: "0139b7503642ece9fda513cddd282f25"
server_key_version: "encryptionkey@0"
I20260620 06:10:50.564463 31721 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781935603120653-11672-0/minicluster-data/ts-2/wal/instance:
uuid: "80262711a64f406aaabbeb5458210619"
format_stamp: "Formatted at 2026-06-20 06:10:50 on dist-test-slave-9hm5"
server_key: "704eab533512bf9e29f13577b72aec22"
server_key_iv: "0139b7503642ece9fda513cddd282f25"
server_key_version: "encryptionkey@0"
I20260620 06:10:50.568219 31721 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260620 06:10:50.570554 31735 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:10:50.571776 31721 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 06:10:50.571894 31721 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781935603120653-11672-0/minicluster-data/ts-2/data,/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781935603120653-11672-0/minicluster-data/ts-2/wal
uuid: "80262711a64f406aaabbeb5458210619"
format_stamp: "Formatted at 2026-06-20 06:10:50 on dist-test-slave-9hm5"
server_key: "704eab533512bf9e29f13577b72aec22"
server_key_iv: "0139b7503642ece9fda513cddd282f25"
server_key_version: "encryptionkey@0"
I20260620 06:10:50.572010 31721 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781935603120653-11672-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781935603120653-11672-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781935603120653-11672-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:10:50.599931 31721 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:10:50.600657 31721 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:10:50.600870 31721 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:10:50.601493 31721 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 06:10:50.602484 31721 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 06:10:50.602545 31721 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 06:10:50.602635 31721 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 06:10:50.602682 31721 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 06:10:50.612638 31721 rpc_server.cc:307] RPC server started. Bound to: 127.11.102.3:34615
I20260620 06:10:50.612663 31848 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.102.3:34615 every 8 connection(s)
I20260620 06:10:50.613554 31721 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781935603120653-11672-0/minicluster-data/ts-2/data/info.pb
I20260620 06:10:50.622504 11672 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9HMp43/build/debug/bin/kudu as pid 31721
I20260620 06:10:50.622638 11672 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781935603120653-11672-0/minicluster-data/ts-2/wal/instance
I20260620 06:10:50.622879 31438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:10:50.615364 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.102.3:36689 (local address 127.11.102.62:36325)
0620 06:10:50.615548 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0620 06:10:50.615552 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 06:10:50.616677 (+ 1125us) server_negotiation.cc:407] Connection header received
0620 06:10:50.618211 (+ 1534us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:10:50.618214 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:10:50.618270 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:10:50.618375 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:10:50.619309 (+ 934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:10:50.619900 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:10:50.621019 (+ 1119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:10:50.621253 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:10:50.621365 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:10:50.622365 (+ 1000us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:10:50.622374 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:10:50.622376 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 06:10:50.622387 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:10:50.622407 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 06:10:50.622415 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:10:50.622487 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0620 06:10:50.622671 (+ 184us) server_negotiation.cc:299] Negotiation successful
0620 06:10:50.622733 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":96}
I20260620 06:10:50.622972 11672 external_mini_cluster.cc:1428] Setting key 5a6481791f3895b403db1f5d9d00c608
I20260620 06:10:50.623826 31851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:10:50.615799 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.102.62:36325 (local address 127.11.102.3:36689)
0620 06:10:50.616511 (+ 712us) negotiation.cc:107] Waiting for socket to connect
0620 06:10:50.616552 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0620 06:10:50.618006 (+ 1454us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:10:50.618548 (+ 542us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:10:50.618555 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:10:50.618688 (+ 133us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:10:50.619156 (+ 468us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:10:50.619169 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:10:50.620028 (+ 859us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:10:50.620031 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:10:50.620896 (+ 865us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:10:50.620905 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:10:50.621400 (+ 495us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:10:50.621404 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:10:50.621564 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:10:50.622116 (+ 552us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 06:10:50.622117 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:10:50.622154 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 06:10:50.622212 (+ 58us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:10:50.622500 (+ 288us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:10:50.622512 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:10:50.622521 (+ 9us) client_negotiation.cc:769] Sending connection context
0620 06:10:50.622657 (+ 136us) client_negotiation.cc:240] Negotiation successful
0620 06:10:50.622916 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":290,"thread_start_us":114,"threads_started":1}
I20260620 06:10:50.624878 31849 heartbeater.cc:347] Connected to a master server at 127.11.102.62:36325
I20260620 06:10:50.625101 31849 heartbeater.cc:464] Registering TS with master...
I20260620 06:10:50.625618 31849 heartbeater.cc:511] Master 127.11.102.62:36325 requested a full tablet report, sending...
I20260620 06:10:50.626624 31398 ts_manager.cc:194] Registered new tserver with Master: 80262711a64f406aaabbeb5458210619 (127.11.102.3:34615)
I20260620 06:10:50.627177 31398 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.11.102.3:36689
I20260620 06:10:50.637169 11672 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 06:10:50.642761 31438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:10:50.639196 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45776 (local address 127.11.102.62:36325)
0620 06:10:50.639343 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0620 06:10:50.639346 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 06:10:50.639524 (+ 178us) server_negotiation.cc:407] Connection header received
0620 06:10:50.639679 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:10:50.639681 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:10:50.639727 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:10:50.639797 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:10:50.640112 (+ 315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:10:50.640617 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:10:50.641392 (+ 775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:10:50.641600 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:10:50.641675 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:10:50.642413 (+ 738us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:10:50.642416 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:10:50.642418 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 06:10:50.642429 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:10:50.642444 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 06:10:50.642452 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:10:50.642509 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0620 06:10:50.642601 (+ 92us) server_negotiation.cc:299] Negotiation successful
0620 06:10:50.642649 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260620 06:10:50.645668 31398 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45776:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 06:10:50.648635 31398 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 06:10:50.662839 31863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:10:50.656689 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.102.1:44483 (local address 127.0.0.1:59646)
0620 06:10:50.657544 (+ 855us) negotiation.cc:107] Waiting for socket to connect
0620 06:10:50.657597 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0620 06:10:50.657727 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:10:50.658593 (+ 866us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:10:50.658596 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:10:50.658613 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 06:10:50.658709 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:10:50.658715 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:10:50.660051 (+ 1336us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:10:50.660055 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:10:50.661535 (+ 1480us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:10:50.661546 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:10:50.662313 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:10:50.662316 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:10:50.662418 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:10:50.662453 (+ 35us) client_negotiation.cc:769] Sending connection context
0620 06:10:50.662577 (+ 124us) client_negotiation.cc:240] Negotiation successful
0620 06:10:50.662692 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":710,"mutex_wait_us":4,"thread_start_us":99,"threads_started":1}
I20260620 06:10:50.662847 31865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:10:50.657334 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59646 (local address 127.11.102.1:44483)
0620 06:10:50.658138 (+ 804us) server_negotiation.cc:206] Beginning negotiation
0620 06:10:50.658150 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 06:10:50.658170 (+ 20us) server_negotiation.cc:407] Connection header received
0620 06:10:50.658265 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:10:50.658270 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:10:50.658446 (+ 176us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:10:50.658606 (+ 160us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 06:10:50.658821 (+ 215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:10:50.659957 (+ 1136us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:10:50.661679 (+ 1722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:10:50.662190 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:10:50.662313 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:10:50.662494 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0620 06:10:50.662604 (+ 110us) server_negotiation.cc:299] Negotiation successful
0620 06:10:50.662691 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":702,"thread_start_us":130,"threads_started":1}
I20260620 06:10:50.663054 31860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:10:50.656523 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.102.3:34615 (local address 127.0.0.1:44180)
0620 06:10:50.657775 (+ 1252us) negotiation.cc:107] Waiting for socket to connect
0620 06:10:50.657790 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 06:10:50.657873 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:10:50.658347 (+ 474us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:10:50.658353 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:10:50.658375 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 06:10:50.658504 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:10:50.658514 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:10:50.660000 (+ 1486us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:10:50.660004 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:10:50.661359 (+ 1355us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:10:50.661368 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:10:50.662614 (+ 1246us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:10:50.662618 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:10:50.662709 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:10:50.662732 (+ 23us) client_negotiation.cc:769] Sending connection context
0620 06:10:50.662805 (+ 73us) client_negotiation.cc:240] Negotiation successful
0620 06:10:50.662885 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":630,"mutex_wait_us":195,"thread_start_us":75,"threads_started":1}
I20260620 06:10:50.663326 31861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:10:50.657173 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.102.2:33177 (local address 127.0.0.1:57724)
0620 06:10:50.657544 (+ 371us) negotiation.cc:107] Waiting for socket to connect
0620 06:10:50.657597 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0620 06:10:50.657727 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:10:50.659016 (+ 1289us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:10:50.659019 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:10:50.659035 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 06:10:50.659128 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:10:50.659136 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:10:50.660294 (+ 1158us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:10:50.660297 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:10:50.661537 (+ 1240us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:10:50.661552 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:10:50.662663 (+ 1111us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:10:50.662666 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:10:50.662770 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:10:50.662802 (+ 32us) client_negotiation.cc:769] Sending connection context
0620 06:10:50.662859 (+ 57us) client_negotiation.cc:240] Negotiation successful
0620 06:10:50.663212 (+ 353us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":257,"mutex_wait_us":140,"thread_start_us":95,"threads_started":1}
I20260620 06:10:50.663702 31862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:10:50.656693 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44180 (local address 127.11.102.3:34615)
0620 06:10:50.657063 (+ 370us) server_negotiation.cc:206] Beginning negotiation
0620 06:10:50.657075 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 06:10:50.657881 (+ 806us) server_negotiation.cc:407] Connection header received
0620 06:10:50.657967 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:10:50.657972 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:10:50.658204 (+ 232us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:10:50.658348 (+ 144us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 06:10:50.658633 (+ 285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:10:50.659878 (+ 1245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:10:50.661483 (+ 1605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:10:50.662262 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:10:50.662369 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:10:50.662550 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0620 06:10:50.663259 (+ 709us) server_negotiation.cc:299] Negotiation successful
0620 06:10:50.663354 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":258,"thread_start_us":149,"threads_started":1}
I20260620 06:10:50.664463 31864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:10:50.657325 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57724 (local address 127.11.102.2:33177)
0620 06:10:50.658621 (+ 1296us) server_negotiation.cc:206] Beginning negotiation
0620 06:10:50.658632 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 06:10:50.658652 (+ 20us) server_negotiation.cc:407] Connection header received
0620 06:10:50.658730 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:10:50.658737 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:10:50.658897 (+ 160us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:10:50.659036 (+ 139us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 06:10:50.659270 (+ 234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:10:50.660175 (+ 905us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:10:50.661664 (+ 1489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:10:50.662390 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:10:50.662478 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:10:50.662615 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0620 06:10:50.663070 (+ 455us) server_negotiation.cc:299] Negotiation successful
0620 06:10:50.663567 (+ 497us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1176,"thread_start_us":144,"threads_started":1}
I20260620 06:10:50.666131 31521 tablet_service.cc:1467] Processing CreateTablet for tablet 59c4a0f55fca4e9dbcf5fb7355a70ccc (DEFAULT_TABLE table=test-table [id=e420b9499420481e8c00c0ef41c72f91]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 06:10:50.666257 31783 tablet_service.cc:1467] Processing CreateTablet for tablet 59c4a0f55fca4e9dbcf5fb7355a70ccc (DEFAULT_TABLE table=test-table [id=e420b9499420481e8c00c0ef41c72f91]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 06:10:50.667091 31521 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 59c4a0f55fca4e9dbcf5fb7355a70ccc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 06:10:50.667239 31783 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 59c4a0f55fca4e9dbcf5fb7355a70ccc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 06:10:50.666901 31652 tablet_service.cc:1467] Processing CreateTablet for tablet 59c4a0f55fca4e9dbcf5fb7355a70ccc (DEFAULT_TABLE table=test-table [id=e420b9499420481e8c00c0ef41c72f91]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 06:10:50.667977 31652 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 59c4a0f55fca4e9dbcf5fb7355a70ccc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 06:10:50.673080 31866 tablet_bootstrap.cc:492] T 59c4a0f55fca4e9dbcf5fb7355a70ccc P 0ba8283540404efd8c98de39a4a269ae: Bootstrap starting.
I20260620 06:10:50.674347 31868 tablet_bootstrap.cc:492] T 59c4a0f55fca4e9dbcf5fb7355a70ccc P 2202052527374fbab20b488c9fec4d7d: Bootstrap starting.
I20260620 06:10:50.675038 31867 tablet_bootstrap.cc:492] T 59c4a0f55fca4e9dbcf5fb7355a70ccc P 80262711a64f406aaabbeb5458210619: Bootstrap starting.
I20260620 06:10:50.675532 31866 tablet_bootstrap.cc:654] T 59c4a0f55fca4e9dbcf5fb7355a70ccc P 0ba8283540404efd8c98de39a4a269ae: Neither blocks nor log segments found. Creating new log.
I20260620 06:10:50.676479 31868 tablet_bootstrap.cc:654] T 59c4a0f55fca4e9dbcf5fb7355a70ccc P 2202052527374fbab20b488c9fec4d7d: Neither blocks nor log segments found. Creating new log.
I20260620 06:10:50.676479 31866 log.cc:824] T 59c4a0f55fca4e9dbcf5fb7355a70ccc P 0ba8283540404efd8c98de39a4a269ae: Log is configured to *not* fsync() on all Append() calls
I20260620 06:10:50.677229 31867 tablet_bootstrap.cc:654] T 59c4a0f55fca4e9dbcf5fb7355a70ccc P 80262711a64f406aaabbeb5458210619: Neither blocks nor log segments found. Creating new log.
I20260620 06:10:50.677289 31868 log.cc:824] T 59c4a0f55fca4e9dbcf5fb7355a70ccc P 2202052527374fbab20b488c9fec4d7d: Log is configured to *not* fsync() on all Append() calls
I20260620 06:10:50.677968 31867 log.cc:824] T 59c4a0f55fca4e9dbcf5fb7355a70ccc P 80262711a64f406aaabbeb5458210619: Log is configured to *not* fsync() on all Append() calls
I20260620 06:10:50.678472 31866 tablet_bootstrap.cc:492] T 59c4a0f55fca4e9dbcf5fb7355a70ccc P 0ba8283540404efd8c98de39a4a269ae: No bootstrap required, opened a new log
I20260620 06:10:50.678663 31866 ts_tablet_manager.cc:1397] T 59c4a0f55fca4e9dbcf5fb7355a70ccc P 0ba8283540404efd8c98de39a4a269ae: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260620 06:10:50.679271 31868 tablet_bootstrap.cc:492] T 59c4a0f55fca4e9dbcf5fb7355a70ccc P 2202052527374fbab20b488c9fec4d7d: No bootstrap required, opened a new log
I20260620 06:10:50.679446 31868 ts_tablet_manager.cc:1397] T 59c4a0f55fca4e9dbcf5fb7355a70ccc P 2202052527374fbab20b488c9fec4d7d: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260620 06:10:50.679646 31867 tablet_bootstrap.cc:492] T 59c4a0f55fca4e9dbcf5fb7355a70ccc P 80262711a64f406aaabbeb5458210619: No bootstrap required, opened a new log
I20260620 06:10:50.679811 31867 ts_tablet_manager.cc:1397] T 59c4a0f55fca4e9dbcf5fb7355a70ccc P 80262711a64f406aaabbeb5458210619: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260620 06:10:50.682989 31867 raft_consensus.cc:348] T 59c4a0f55fca4e9dbcf5fb7355a70ccc P 80262711a64f406aaabbeb5458210619 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80262711a64f406aaabbeb5458210619" member_type: VOTER last_known_addr { host: "127.11.102.3" port: 34615 } } peers { permanent_uuid: "0ba8283540404efd8c98de39a4a269ae" member_type: VOTER last_known_addr { host: "127.11.102.1" port: 44483 } } peers { permanent_uuid: "2202052527374fbab20b488c9fec4d7d" member_type: VOTER last_known_addr { host: "127.11.102.2" port: 33177 } }
I20260620 06:10:50.682983 31866 raft_consensus.cc:348] T 59c4a0f55fca4e9dbcf5fb7355a70ccc P 0ba8283540404efd8c98de39a4a269ae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80262711a64f406aaabbeb5458210619" member_type: VOTER last_known_addr { host: "127.11.102.3" port: 34615 } } peers { permanent_uuid: "0ba8283540404efd8c98de39a4a269ae" member_type: VOTER last_known_addr { host: "127.11.102.1" port: 44483 } } peers { permanent_uuid: "2202052527374fbab20b488c9fec4d7d" member_type: VOTER last_known_addr { host: "127.11.102.2" port: 33177 } }
I20260620 06:10:50.683219 31867 raft_consensus.cc:374] T 59c4a0f55fca4e9dbcf5fb7355a70ccc P 80262711a64f406aaabbeb5458210619 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 06:10:50.683220 31866 raft_consensus.cc:374] T 59c4a0f55fca4e9dbcf5fb7355a70ccc P 0ba8283540404efd8c98de39a4a269ae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 06:10:50.683271 31866 raft_consensus.cc:729] T 59c4a0f55fca4e9dbcf5fb7355a70ccc P 0ba8283540404efd8c98de39a4a269ae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ba8283540404efd8c98de39a4a269ae, State: Initialized, Role: FOLLOWER
I20260620 06:10:50.683270 31867 raft_consensus.cc:729] T 59c4a0f55fca4e9dbcf5fb7355a70ccc P 80262711a64f406aaabbeb5458210619 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 80262711a64f406aaabbeb5458210619, State: Initialized, Role: FOLLOWER
I20260620 06:10:50.683431 31868 raft_consensus.cc:348] T 59c4a0f55fca4e9dbcf5fb7355a70ccc P 2202052527374fbab20b488c9fec4d7d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80262711a64f406aaabbeb5458210619" member_type: VOTER last_known_addr { host: "127.11.102.3" port: 34615 } } peers { permanent_uuid: "0ba8283540404efd8c98de39a4a269ae" member_type: VOTER last_known_addr { host: "127.11.102.1" port: 44483 } } peers { permanent_uuid: "2202052527374fbab20b488c9fec4d7d" member_type: VOTER last_known_addr { host: "127.11.102.2" port: 33177 } }
I20260620 06:10:50.683661 31868 raft_consensus.cc:374] T 59c4a0f55fca4e9dbcf5fb7355a70ccc P 2202052527374fbab20b488c9fec4d7d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 06:10:50.683713 31868 raft_consensus.cc:729] T 59c4a0f55fca4e9dbcf5fb7355a70ccc P 2202052527374fbab20b488c9fec4d7d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2202052527374fbab20b488c9fec4d7d, State: Initialized, Role: FOLLOWER
I20260620 06:10:50.683727 31867 consensus_queue.cc:260] T 59c4a0f55fca4e9dbcf5fb7355a70ccc P 80262711a64f406aaabbeb5458210619 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80262711a64f406aaabbeb5458210619" member_type: VOTER last_known_addr { host: "127.11.102.3" port: 34615 } } peers { permanent_uuid: "0ba8283540404efd8c98de39a4a269ae" member_type: VOTER last_known_addr { host: "127.11.102.1" port: 44483 } } peers { permanent_uuid: "2202052527374fbab20b488c9fec4d7d" member_type: VOTER last_known_addr { host: "127.11.102.2" port: 33177 } }
I20260620 06:10:50.683734 31866 consensus_queue.cc:260] T 59c4a0f55fca4e9dbcf5fb7355a70ccc P 0ba8283540404efd8c98de39a4a269ae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80262711a64f406aaabbeb5458210619" member_type: VOTER last_known_addr { host: "127.11.102.3" port: 34615 } } peers { permanent_uuid: "0ba8283540404efd8c98de39a4a269ae" member_type: VOTER last_known_addr { host: "127.11.102.1" port: 44483 } } peers { permanent_uuid: "2202052527374fbab20b488c9fec4d7d" member_type: VOTER last_known_addr { host: "127.11.102.2" port: 33177 } }
I20260620 06:10:50.684350 31849 heartbeater.cc:503] Master 127.11.102.62:36325 was elected leader, sending a full tablet report...
I20260620 06:10:50.684715 31866 ts_tablet_manager.cc:1428] T 59c4a0f55fca4e9dbcf5fb7355a70ccc P 0ba8283540404efd8c98de39a4a269ae: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260620 06:10:50.684717 31867 ts_tablet_manager.cc:1428] T 59c4a0f55fca4e9dbcf5fb7355a70ccc P 80262711a64f406aaabbeb5458210619: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20260620 06:10:50.685046 31587 heartbeater.cc:503] Master 127.11.102.62:36325 was elected leader, sending a full tablet report...
I20260620 06:10:50.684159 31868 consensus_queue.cc:260] T 59c4a0f55fca4e9dbcf5fb7355a70ccc P 2202052527374fbab20b488c9fec4d7d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80262711a64f406aaabbeb5458210619" member_type: VOTER last_known_addr { host: "127.11.102.3" port: 34615 } } peers { permanent_uuid: "0ba8283540404efd8c98de39a4a269ae" member_type: VOTER last_known_addr { host: "127.11.102.1" port: 44483 } } peers { permanent_uuid: "2202052527374fbab20b488c9fec4d7d" member_type: VOTER last_known_addr { host: "127.11.102.2" port: 33177 } }
I20260620 06:10:50.685758 31718 heartbeater.cc:503] Master 127.11.102.62:36325 was elected leader, sending a full tablet report...
I20260620 06:10:50.686018 31868 ts_tablet_manager.cc:1428] T 59c4a0f55fca4e9dbcf5fb7355a70ccc P 2202052527374fbab20b488c9fec4d7d: Time spent starting tablet: real 0.007s user 0.000s sys 0.006s
W20260620 06:10:50.732162 31588 tablet.cc:2369] T 59c4a0f55fca4e9dbcf5fb7355a70ccc P 0ba8283540404efd8c98de39a4a269ae: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 06:10:50.828346 31872 raft_consensus.cc:482] T 59c4a0f55fca4e9dbcf5fb7355a70ccc P 80262711a64f406aaabbeb5458210619 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 06:10:50.828596 31872 raft_consensus.cc:504] T 59c4a0f55fca4e9dbcf5fb7355a70ccc P 80262711a64f406aaabbeb5458210619 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80262711a64f406aaabbeb5458210619" member_type: VOTER last_known_addr { host: "127.11.102.3" port: 34615 } } peers { permanent_uuid: "0ba8283540404efd8c98de39a4a269ae" member_type: VOTER last_known_addr { host: "127.11.102.1" port: 44483 } } peers { permanent_uuid: "2202052527374fbab20b488c9fec4d7d" member_type: VOTER last_known_addr { host: "127.11.102.2" port: 33177 } }
I20260620 06:10:50.829764 31872 leader_election.cc:288] T 59c4a0f55fca4e9dbcf5fb7355a70ccc P 80262711a64f406aaabbeb5458210619 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0ba8283540404efd8c98de39a4a269ae (127.11.102.1:44483), 2202052527374fbab20b488c9fec4d7d (127.11.102.2:33177)
I20260620 06:10:50.834537 31864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:10:50.830159 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.102.3:36345 (local address 127.11.102.2:33177)
0620 06:10:50.830292 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0620 06:10:50.830297 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 06:10:50.830802 (+ 505us) server_negotiation.cc:407] Connection header received
0620 06:10:50.830899 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:10:50.830902 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:10:50.830951 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:10:50.831033 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 06:10:50.831274 (+ 241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:10:50.831946 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:10:50.833293 (+ 1347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:10:50.833853 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:10:50.833956 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:10:50.833993 (+ 37us) server_negotiation.cc:1035] Waiting for connection context
0620 06:10:50.834330 (+ 337us) server_negotiation.cc:299] Negotiation successful
0620 06:10:50.834401 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260620 06:10:50.834548 31851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:10:50.830033 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.102.1:44483 (local address 127.11.102.3:52053)
0620 06:10:50.830181 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0620 06:10:50.830198 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 06:10:50.830308 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:10:50.830606 (+ 298us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:10:50.830609 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:10:50.830626 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 06:10:50.830777 (+ 151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:10:50.830786 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:10:50.831822 (+ 1036us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:10:50.831825 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:10:50.833045 (+ 1220us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:10:50.833055 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:10:50.833943 (+ 888us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:10:50.833949 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:10:50.834150 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:10:50.834217 (+ 67us) client_negotiation.cc:769] Sending connection context
0620 06:10:50.834293 (+ 76us) client_negotiation.cc:240] Negotiation successful
0620 06:10:50.834388 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":52,"mutex_wait_us":87}
I20260620 06:10:50.834548 31865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:10:50.830148 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.11.102.3:52053 (local address 127.11.102.1:44483)
0620 06:10:50.830293 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0620 06:10:50.830298 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 06:10:50.830315 (+ 17us) server_negotiation.cc:407] Connection header received
0620 06:10:50.830422 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:10:50.830425 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:10:50.830487 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:10:50.830604 (+ 117us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 06:10:50.830898 (+ 294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:10:50.831712 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:10:50.833179 (+ 1467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:10:50.833801 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:10:50.833941 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:10:50.833992 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0620 06:10:50.834330 (+ 338us) server_negotiation.cc:299] Negotiation successful
0620 06:10:50.834401 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260620 06:10:50.834556 31875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:10:50.830033 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.102.2:33177 (local address 127.11.102.3:36345)
0620 06:10:50.830696 (+ 663us) negotiation.cc:107] Waiting for socket to connect
0620 06:10:50.830710 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 06:10:50.830802 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:10:50.831049 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:10:50.831051 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:10:50.831064 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 06:10:50.831160 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:10:50.831165 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:10:50.832050 (+ 885us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:10:50.832053 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:10:50.833164 (+ 1111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:10:50.833173 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:10:50.833965 (+ 792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:10:50.833968 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:10:50.834153 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:10:50.834215 (+ 62us) client_negotiation.cc:769] Sending connection context
0620 06:10:50.834293 (+ 78us) client_negotiation.cc:240] Negotiation successful
0620 06:10:50.834388 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":592,"mutex_wait_us":358,"thread_start_us":131,"threads_started":1}
I20260620 06:10:50.835305 31541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59c4a0f55fca4e9dbcf5fb7355a70ccc" candidate_uuid: "80262711a64f406aaabbeb5458210619" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ba8283540404efd8c98de39a4a269ae" is_pre_election: true
I20260620 06:10:50.835507 31672 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59c4a0f55fca4e9dbcf5fb7355a70ccc" candidate_uuid: "80262711a64f406aaabbeb5458210619" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2202052527374fbab20b488c9fec4d7d" is_pre_election: true
I20260620 06:10:50.835660 31541 raft_consensus.cc:2454] T 59c4a0f55fca4e9dbcf5fb7355a70ccc P 0ba8283540404efd8c98de39a4a269ae [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 80262711a64f406aaabbeb5458210619 in term 0.
I20260620 06:10:50.835827 31672 raft_consensus.cc:2454] T 59c4a0f55fca4e9dbcf5fb7355a70ccc P 2202052527374fbab20b488c9fec4d7d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 80262711a64f406aaabbeb5458210619 in term 0.
I20260620 06:10:50.836172 31737 leader_election.cc:302] T 59c4a0f55fca4e9dbcf5fb7355a70ccc P 80262711a64f406aaabbeb5458210619 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 0ba8283540404efd8c98de39a4a269ae, 2202052527374fbab20b488c9fec4d7d, 80262711a64f406aaabbeb5458210619; no voters:
I20260620 06:10:50.836452 31872 raft_consensus.cc:2781] T 59c4a0f55fca4e9dbcf5fb7355a70ccc P 80262711a64f406aaabbeb5458210619 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 06:10:50.836549 31872 raft_consensus.cc:482] T 59c4a0f55fca4e9dbcf5fb7355a70ccc P 80262711a64f406aaabbeb5458210619 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 06:10:50.836609 31872 raft_consensus.cc:3037] T 59c4a0f55fca4e9dbcf5fb7355a70ccc P 80262711a64f406aaabbeb5458210619 [term 0 FOLLOWER]: Advancing to term 1
I20260620 06:10:50.837612 31872 raft_consensus.cc:504] T 59c4a0f55fca4e9dbcf5fb7355a70ccc P 80262711a64f406aaabbeb5458210619 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80262711a64f406aaabbeb5458210619" member_type: VOTER last_known_addr { host: "127.11.102.3" port: 34615 } } peers { permanent_uuid: "0ba8283540404efd8c98de39a4a269ae" member_type: VOTER last_known_addr { host: "127.11.102.1" port: 44483 } } peers { permanent_uuid: "2202052527374fbab20b488c9fec4d7d" member_type: VOTER last_known_addr { host: "127.11.102.2" port: 33177 } }
I20260620 06:10:50.838056 31872 leader_election.cc:288] T 59c4a0f55fca4e9dbcf5fb7355a70ccc P 80262711a64f406aaabbeb5458210619 [CANDIDATE]: Term 1 election: Requested vote from peers 0ba8283540404efd8c98de39a4a269ae (127.11.102.1:44483), 2202052527374fbab20b488c9fec4d7d (127.11.102.2:33177)
I20260620 06:10:50.838380 31672 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59c4a0f55fca4e9dbcf5fb7355a70ccc" candidate_uuid: "80262711a64f406aaabbeb5458210619" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2202052527374fbab20b488c9fec4d7d"
I20260620 06:10:50.838395 31541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59c4a0f55fca4e9dbcf5fb7355a70ccc" candidate_uuid: "80262711a64f406aaabbeb5458210619" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ba8283540404efd8c98de39a4a269ae"
I20260620 06:10:50.838498 31672 raft_consensus.cc:3037] T 59c4a0f55fca4e9dbcf5fb7355a70ccc P 2202052527374fbab20b488c9fec4d7d [term 0 FOLLOWER]: Advancing to term 1
I20260620 06:10:50.838498 31541 raft_consensus.cc:3037] T 59c4a0f55fca4e9dbcf5fb7355a70ccc P 0ba8283540404efd8c98de39a4a269ae [term 0 FOLLOWER]: Advancing to term 1
I20260620 06:10:50.839788 31672 raft_consensus.cc:2454] T 59c4a0f55fca4e9dbcf5fb7355a70ccc P 2202052527374fbab20b488c9fec4d7d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 80262711a64f406aaabbeb5458210619 in term 1.
I20260620 06:10:50.839788 31541 raft_consensus.cc:2454] T 59c4a0f55fca4e9dbcf5fb7355a70ccc P 0ba8283540404efd8c98de39a4a269ae [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 80262711a64f406aaabbeb5458210619 in term 1.
I20260620 06:10:50.840097 31737 leader_election.cc:302] T 59c4a0f55fca4e9dbcf5fb7355a70ccc P 80262711a64f406aaabbeb5458210619 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ba8283540404efd8c98de39a4a269ae, 80262711a64f406aaabbeb5458210619; no voters:
I20260620 06:10:50.840271 31872 raft_consensus.cc:2781] T 59c4a0f55fca4e9dbcf5fb7355a70ccc P 80262711a64f406aaabbeb5458210619 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 06:10:50.840469 31872 raft_consensus.cc:686] T 59c4a0f55fca4e9dbcf5fb7355a70ccc P 80262711a64f406aaabbeb5458210619 [term 1 LEADER]: Becoming Leader. State: Replica: 80262711a64f406aaabbeb5458210619, State: Running, Role: LEADER
I20260620 06:10:50.840791 31872 consensus_queue.cc:237] T 59c4a0f55fca4e9dbcf5fb7355a70ccc P 80262711a64f406aaabbeb5458210619 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80262711a64f406aaabbeb5458210619" member_type: VOTER last_known_addr { host: "127.11.102.3" port: 34615 } } peers { permanent_uuid: "0ba8283540404efd8c98de39a4a269ae" member_type: VOTER last_known_addr { host: "127.11.102.1" port: 44483 } } peers { permanent_uuid: "2202052527374fbab20b488c9fec4d7d" member_type: VOTER last_known_addr { host: "127.11.102.2" port: 33177 } }
I20260620 06:10:50.844058 31398 catalog_manager.cc:5495] T 59c4a0f55fca4e9dbcf5fb7355a70ccc P 80262711a64f406aaabbeb5458210619 reported cstate change: term changed from 0 to 1, leader changed from <none> to 80262711a64f406aaabbeb5458210619 (127.11.102.3). New cstate: current_term: 1 leader_uuid: "80262711a64f406aaabbeb5458210619" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80262711a64f406aaabbeb5458210619" member_type: VOTER last_known_addr { host: "127.11.102.3" port: 34615 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0ba8283540404efd8c98de39a4a269ae" member_type: VOTER last_known_addr { host: "127.11.102.1" port: 44483 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2202052527374fbab20b488c9fec4d7d" member_type: VOTER last_known_addr { host: "127.11.102.2" port: 33177 } health_report { overall_health: UNKNOWN } } }
W20260620 06:10:50.864457 31850 tablet.cc:2369] T 59c4a0f55fca4e9dbcf5fb7355a70ccc P 80262711a64f406aaabbeb5458210619: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 06:10:50.902318 31862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:10:50.897830 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44196 (local address 127.11.102.3:34615)
0620 06:10:50.898001 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0620 06:10:50.898008 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 06:10:50.898027 (+ 19us) server_negotiation.cc:407] Connection header received
0620 06:10:50.898203 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:10:50.898209 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:10:50.898292 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:10:50.898415 (+ 123us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 06:10:50.898678 (+ 263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:10:50.899189 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:10:50.900270 (+ 1081us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:10:50.900673 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:10:50.900791 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:10:50.901150 (+ 359us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 06:10:50.901761 (+ 611us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 06:10:50.901905 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0620 06:10:50.902022 (+ 117us) server_negotiation.cc:299] Negotiation successful
0620 06:10:50.902119 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260620 06:10:50.909708 31672 raft_consensus.cc:1261] T 59c4a0f55fca4e9dbcf5fb7355a70ccc P 2202052527374fbab20b488c9fec4d7d [term 1 FOLLOWER]: Refusing update from remote peer 80262711a64f406aaabbeb5458210619: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 06:10:50.909708 31541 raft_consensus.cc:1261] T 59c4a0f55fca4e9dbcf5fb7355a70ccc P 0ba8283540404efd8c98de39a4a269ae [term 1 FOLLOWER]: Refusing update from remote peer 80262711a64f406aaabbeb5458210619: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 06:10:50.910425 31876 consensus_queue.cc:1035] T 59c4a0f55fca4e9dbcf5fb7355a70ccc P 80262711a64f406aaabbeb5458210619 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ba8283540404efd8c98de39a4a269ae" member_type: VOTER last_known_addr { host: "127.11.102.1" port: 44483 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 06:10:50.910894 31876 consensus_queue.cc:1035] T 59c4a0f55fca4e9dbcf5fb7355a70ccc P 80262711a64f406aaabbeb5458210619 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2202052527374fbab20b488c9fec4d7d" member_type: VOTER last_known_addr { host: "127.11.102.2" port: 33177 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260620 06:10:50.916057 31719 tablet.cc:2369] T 59c4a0f55fca4e9dbcf5fb7355a70ccc P 2202052527374fbab20b488c9fec4d7d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 06:10:50.920290 31880 mvcc.cc:205] Tried to move back new op lower bound from 7298809245318250496 to 7298809245046476800. Current Snapshot: MvccSnapshot[applied={T|T < 7298809245318250496}]
I20260620 06:10:50.922258 31881 mvcc.cc:205] Tried to move back new op lower bound from 7298809245318250496 to 7298809245046476800. Current Snapshot: MvccSnapshot[applied={T|T < 7298809245318250496 or (T in {7298809245318250496})}]
I20260620 06:10:50.929014 31397 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:45776:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 06:10:50.929193 31397 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:45776:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 06:10:50.931602 31397 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 286eba4a81ba42e7b96d742071c0a075: Sending DeleteTablet for 3 replicas of tablet 59c4a0f55fca4e9dbcf5fb7355a70ccc
I20260620 06:10:50.932379 31783 tablet_service.cc:1514] Processing DeleteTablet for tablet 59c4a0f55fca4e9dbcf5fb7355a70ccc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 06:10:50 UTC) from {username='slave'} at 127.0.0.1:44180
I20260620 06:10:50.932530 31521 tablet_service.cc:1514] Processing DeleteTablet for tablet 59c4a0f55fca4e9dbcf5fb7355a70ccc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 06:10:50 UTC) from {username='slave'} at 127.0.0.1:59646
I20260620 06:10:50.932682 31652 tablet_service.cc:1514] Processing DeleteTablet for tablet 59c4a0f55fca4e9dbcf5fb7355a70ccc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 06:10:50 UTC) from {username='slave'} at 127.0.0.1:57724
I20260620 06:10:50.933085 31891 tablet_replica.cc:330] T 59c4a0f55fca4e9dbcf5fb7355a70ccc P 80262711a64f406aaabbeb5458210619: stopping tablet replica
I20260620 06:10:50.933241 31893 tablet_replica.cc:330] T 59c4a0f55fca4e9dbcf5fb7355a70ccc P 2202052527374fbab20b488c9fec4d7d: stopping tablet replica
I20260620 06:10:50.933367 31891 raft_consensus.cc:2229] T 59c4a0f55fca4e9dbcf5fb7355a70ccc P 80262711a64f406aaabbeb5458210619 [term 1 LEADER]: Raft consensus shutting down.
I20260620 06:10:50.933382 11672 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9HMp43/build/debug/bin/kudu with pid 31459
I20260620 06:10:50.933718 31891 raft_consensus.cc:2258] T 59c4a0f55fca4e9dbcf5fb7355a70ccc P 80262711a64f406aaabbeb5458210619 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 06:10:50.934093 31893 raft_consensus.cc:2229] T 59c4a0f55fca4e9dbcf5fb7355a70ccc P 2202052527374fbab20b488c9fec4d7d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 06:10:50.934451 31893 raft_consensus.cc:2258] T 59c4a0f55fca4e9dbcf5fb7355a70ccc P 2202052527374fbab20b488c9fec4d7d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 06:10:50.935993 31893 ts_tablet_manager.cc:1905] T 59c4a0f55fca4e9dbcf5fb7355a70ccc P 2202052527374fbab20b488c9fec4d7d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 06:10:50.939149 31893 ts_tablet_manager.cc:1918] T 59c4a0f55fca4e9dbcf5fb7355a70ccc P 2202052527374fbab20b488c9fec4d7d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 06:10:50.939244 31893 log.cc:1196] T 59c4a0f55fca4e9dbcf5fb7355a70ccc P 2202052527374fbab20b488c9fec4d7d: Deleting WAL directory at /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781935603120653-11672-0/minicluster-data/ts-1/wal/wals/59c4a0f55fca4e9dbcf5fb7355a70ccc
I20260620 06:10:50.939582 31893 ts_tablet_manager.cc:1939] T 59c4a0f55fca4e9dbcf5fb7355a70ccc P 2202052527374fbab20b488c9fec4d7d: Deleting consensus metadata
I20260620 06:10:50.939605 31891 ts_tablet_manager.cc:1905] T 59c4a0f55fca4e9dbcf5fb7355a70ccc P 80262711a64f406aaabbeb5458210619: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 06:10:50.940848 31383 catalog_manager.cc:4841] TS 2202052527374fbab20b488c9fec4d7d (127.11.102.2:33177): tablet 59c4a0f55fca4e9dbcf5fb7355a70ccc (table test-table [id=e420b9499420481e8c00c0ef41c72f91]) successfully deleted
I20260620 06:10:50.941953 31891 ts_tablet_manager.cc:1918] T 59c4a0f55fca4e9dbcf5fb7355a70ccc P 80262711a64f406aaabbeb5458210619: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 06:10:50.942046 31891 log.cc:1196] T 59c4a0f55fca4e9dbcf5fb7355a70ccc P 80262711a64f406aaabbeb5458210619: Deleting WAL directory at /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781935603120653-11672-0/minicluster-data/ts-2/wal/wals/59c4a0f55fca4e9dbcf5fb7355a70ccc
I20260620 06:10:50.942364 31891 ts_tablet_manager.cc:1939] T 59c4a0f55fca4e9dbcf5fb7355a70ccc P 80262711a64f406aaabbeb5458210619: Deleting consensus metadata
I20260620 06:10:50.943260 31385 catalog_manager.cc:4841] TS 80262711a64f406aaabbeb5458210619 (127.11.102.3:34615): tablet 59c4a0f55fca4e9dbcf5fb7355a70ccc (table test-table [id=e420b9499420481e8c00c0ef41c72f91]) successfully deleted
W20260620 06:10:50.947216 31384 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.11.102.1:44483 (error 108)
I20260620 06:10:50.947363 11672 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9HMp43/build/debug/bin/kudu with pid 31590
I20260620 06:10:50.949054 31861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:10:50.948744 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.102.1:44483 (local address 127.0.0.1:59662)
0620 06:10:50.948856 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0620 06:10:50.948933 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.102.1:44483: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260620 06:10:50.949218 31384 catalog_manager.cc:4584] TS 0ba8283540404efd8c98de39a4a269ae (127.11.102.1:44483): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 59c4a0f55fca4e9dbcf5fb7355a70ccc: Network error: Client connection negotiation failed: client connection to 127.11.102.1:44483: connect: Connection refused (error 111)
I20260620 06:10:50.952975 11672 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9HMp43/build/debug/bin/kudu with pid 31721
I20260620 06:10:50.958774 11672 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9HMp43/build/debug/bin/kudu with pid 31368
2026-06-20T06:10:50Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (1158 ms)
[----------] 6 tests from EncryptionPolicyTest (10377 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781935603120653-11672-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 06:10:51 dist-test-slave-9hm5 krb5kdc[31897](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 06:10:51 dist-test-slave-9hm5 krb5kdc[31897](info): set up 2 sockets
Jun 20 06:10:51 dist-test-slave-9hm5 krb5kdc[31897](info): commencing operation
krb5kdc: starting...
W20260620 06:10:51.999850 11672 mini_kdc.cc:121] Time spent starting KDC: real 1.020s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781935603120653-11672-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781935603120653-11672-0/krb5kdc/test-user.keytab.
Jun 20 06:10:52 dist-test-slave-9hm5 krb5kdc[31897](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935852, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T06:10:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T06:10:52Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781935603120653-11672-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781935603120653-11672-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 06:10:52.119318 11672 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9HMp43/build/debug/bin/kudu
/tmp/dist-test-task9HMp43/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781935603120653-11672-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781935603120653-11672-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781935603120653-11672-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781935603120653-11672-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43723
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781935603120653-11672-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43039
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43723
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781935603120653-11672-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781935603120653-11672-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781935603120653-11672-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 06:10:52.225869 31910 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:10:52.226183 31910 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:10:52.226255 31910 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:10:52.230320 31910 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 06:10:52.230413 31910 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:10:52.230451 31910 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:10:52.230504 31910 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 06:10:52.230541 31910 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 06:10:52.235870 31910 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43039
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781935603120653-11672-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781935603120653-11672-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43723
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781935603120653-11672-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43723
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781935603120653-11672-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31910
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781935603120653-11672-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 05:28:48 UTC on ccycloud.quasar-fixhms.root.comops.site
build id 4457020
I20260620 06:10:52.236938 31910 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:10:52.237959 31910 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 06:10:52.246471 31915 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:10:52.246515 31918 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:10:52.246665 31910 server_base.cc:1034] running on GCE node
W20260620 06:10:52.246467 31916 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:10:52.247262 31910 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:10:52.248308 31910 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:10:52.249493 31910 hybrid_clock.cc:648] HybridClock initialized: now 1781935852249475 us; error 30 us; skew 500 ppm
Jun 20 06:10:52 dist-test-slave-9hm5 krb5kdc[31897](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935852, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 06:10:52.252118 31910 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 06:10:52.252944 31910 webserver.cc:460] Webserver started at http://127.0.0.1:40333/ using document root <none> and password file <none>
I20260620 06:10:52.253423 31910 fs_manager.cc:359] Metadata directory not provided
I20260620 06:10:52.253479 31910 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:10:52.253641 31910 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 06:10:52.255517 31910 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781935603120653-11672-0/minicluster-data/master-0/data/instance:
uuid: "695f889cbadf4375907d65a29b67108a"
format_stamp: "Formatted at 2026-06-20 06:10:52 on dist-test-slave-9hm5"
server_key: "9b6618d8afbc7330612802dee9761e58"
server_key_iv: "f6671c3ba5012deca82a3325e5448929"
server_key_version: "encryptionkey@0"
I20260620 06:10:52.256086 31910 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781935603120653-11672-0/minicluster-data/master-0/wal/instance:
uuid: "695f889cbadf4375907d65a29b67108a"
format_stamp: "Formatted at 2026-06-20 06:10:52 on dist-test-slave-9hm5"
server_key: "9b6618d8afbc7330612802dee9761e58"
server_key_iv: "f6671c3ba5012deca82a3325e5448929"
server_key_version: "encryptionkey@0"
I20260620 06:10:52.259578 31910 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260620 06:10:52.262001 31925 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:10:52.263202 31910 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 06:10:52.263299 31910 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781935603120653-11672-0/minicluster-data/master-0/data,/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781935603120653-11672-0/minicluster-data/master-0/wal
uuid: "695f889cbadf4375907d65a29b67108a"
format_stamp: "Formatted at 2026-06-20 06:10:52 on dist-test-slave-9hm5"
server_key: "9b6618d8afbc7330612802dee9761e58"
server_key_iv: "f6671c3ba5012deca82a3325e5448929"
server_key_version: "encryptionkey@0"
I20260620 06:10:52.263382 31910 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781935603120653-11672-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781935603120653-11672-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781935603120653-11672-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:10:52.292821 31910 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:10:52.296075 31910 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:10:52.296236 31910 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:10:52.303273 31910 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43723
I20260620 06:10:52.303292 31977 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43723 every 8 connection(s)
I20260620 06:10:52.304280 31910 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781935603120653-11672-0/minicluster-data/master-0/data/info.pb
I20260620 06:10:52.305421 11672 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9HMp43/build/debug/bin/kudu as pid 31910
I20260620 06:10:52.305523 11672 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781935603120653-11672-0/minicluster-data/master-0/wal/instance
I20260620 06:10:52.305835 11672 external_mini_cluster.cc:1428] Setting key b14c32f28596591a4b0228f4c35c3472
I20260620 06:10:52.307394 31978 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 06:10:52.314349 31978 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 695f889cbadf4375907d65a29b67108a: Bootstrap starting.
Jun 20 06:10:52 dist-test-slave-9hm5 krb5kdc[31897](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935852, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260620 06:10:52.316895 31978 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 695f889cbadf4375907d65a29b67108a: Neither blocks nor log segments found. Creating new log.
I20260620 06:10:52.317642 31978 log.cc:824] T 00000000000000000000000000000000 P 695f889cbadf4375907d65a29b67108a: Log is configured to *not* fsync() on all Append() calls
I20260620 06:10:52.319502 31978 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 695f889cbadf4375907d65a29b67108a: No bootstrap required, opened a new log
I20260620 06:10:52.320648 31981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:10:52.307288 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42713 (local address 127.0.0.1:43723)
0620 06:10:52.307734 (+ 446us) server_negotiation.cc:206] Beginning negotiation
0620 06:10:52.307760 (+ 26us) server_negotiation.cc:399] Waiting for connection header
0620 06:10:52.307795 (+ 35us) server_negotiation.cc:407] Connection header received
0620 06:10:52.308635 (+ 840us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:10:52.308662 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:10:52.309160 (+ 498us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:10:52.309471 (+ 311us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:10:52.310430 (+ 959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:10:52.311414 (+ 984us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:10:52.312591 (+ 1177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:10:52.313053 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:10:52.313179 (+ 126us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:10:52.316090 (+ 2911us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:10:52.316095 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:10:52.316108 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:10:52.316134 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:10:52.317874 (+ 1740us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:10:52.318383 (+ 509us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:10:52.318393 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:10:52.318398 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:10:52.318501 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:10:52.318834 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:10:52.318837 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:10:52.318839 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:10:52.319215 (+ 376us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:10:52.319450 (+ 235us) server_negotiation.cc:1035] Waiting for connection context
0620 06:10:52.319753 (+ 303us) server_negotiation.cc:299] Negotiation successful
0620 06:10:52.319966 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":292,"thread_start_us":148,"threads_started":1}
I20260620 06:10:52.323666 11672 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 06:10:52.323658 31978 raft_consensus.cc:348] T 00000000000000000000000000000000 P 695f889cbadf4375907d65a29b67108a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "695f889cbadf4375907d65a29b67108a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43723 } }
I20260620 06:10:52.323939 31978 raft_consensus.cc:374] T 00000000000000000000000000000000 P 695f889cbadf4375907d65a29b67108a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 06:10:52.323997 31978 raft_consensus.cc:729] T 00000000000000000000000000000000 P 695f889cbadf4375907d65a29b67108a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 695f889cbadf4375907d65a29b67108a, State: Initialized, Role: FOLLOWER
I20260620 06:10:52.324427 31978 consensus_queue.cc:260] T 00000000000000000000000000000000 P 695f889cbadf4375907d65a29b67108a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "695f889cbadf4375907d65a29b67108a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43723 } }
I20260620 06:10:52.324568 31978 raft_consensus.cc:388] T 00000000000000000000000000000000 P 695f889cbadf4375907d65a29b67108a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 06:10:52.324630 31978 raft_consensus.cc:482] T 00000000000000000000000000000000 P 695f889cbadf4375907d65a29b67108a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 06:10:52.324730 31978 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 695f889cbadf4375907d65a29b67108a [term 0 FOLLOWER]: Advancing to term 1
I20260620 06:10:52.325644 31978 raft_consensus.cc:504] T 00000000000000000000000000000000 P 695f889cbadf4375907d65a29b67108a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "695f889cbadf4375907d65a29b67108a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43723 } }
I20260620 06:10:52.326040 31978 leader_election.cc:302] T 00000000000000000000000000000000 P 695f889cbadf4375907d65a29b67108a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 695f889cbadf4375907d65a29b67108a; no voters:
I20260620 06:10:52.326421 31978 leader_election.cc:288] T 00000000000000000000000000000000 P 695f889cbadf4375907d65a29b67108a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 06:10:52.326574 31988 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 695f889cbadf4375907d65a29b67108a [term 1 FOLLOWER]: Leader election won for term 1
I20260620 06:10:52.326807 31988 raft_consensus.cc:686] T 00000000000000000000000000000000 P 695f889cbadf4375907d65a29b67108a [term 1 LEADER]: Becoming Leader. State: Replica: 695f889cbadf4375907d65a29b67108a, State: Running, Role: LEADER
I20260620 06:10:52.327095 31988 consensus_queue.cc:237] T 00000000000000000000000000000000 P 695f889cbadf4375907d65a29b67108a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "695f889cbadf4375907d65a29b67108a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43723 } }
I20260620 06:10:52.327790 31978 sys_catalog.cc:564] T 00000000000000000000000000000000 P 695f889cbadf4375907d65a29b67108a [sys.catalog]: configured and running, proceeding with master startup.
I20260620 06:10:52.328624 31989 sys_catalog.cc:455] T 00000000000000000000000000000000 P 695f889cbadf4375907d65a29b67108a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "695f889cbadf4375907d65a29b67108a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "695f889cbadf4375907d65a29b67108a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43723 } } }
I20260620 06:10:52.328735 31989 sys_catalog.cc:458] T 00000000000000000000000000000000 P 695f889cbadf4375907d65a29b67108a [sys.catalog]: This master's current role is: LEADER
I20260620 06:10:52.329108 31990 sys_catalog.cc:455] T 00000000000000000000000000000000 P 695f889cbadf4375907d65a29b67108a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 695f889cbadf4375907d65a29b67108a. Latest consensus state: current_term: 1 leader_uuid: "695f889cbadf4375907d65a29b67108a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "695f889cbadf4375907d65a29b67108a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43723 } } }
I20260620 06:10:52.329272 31990 sys_catalog.cc:458] T 00000000000000000000000000000000 P 695f889cbadf4375907d65a29b67108a [sys.catalog]: This master's current role is: LEADER
I20260620 06:10:52.330008 31997 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 06:10:52.332945 31997 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 06:10:52.334393 31981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:10:52.325818 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48359 (local address 127.0.0.1:43723)
0620 06:10:52.325961 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0620 06:10:52.325964 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 06:10:52.326148 (+ 184us) server_negotiation.cc:407] Connection header received
0620 06:10:52.326284 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:10:52.326287 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:10:52.326325 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:10:52.326417 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:10:52.327200 (+ 783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:10:52.327964 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:10:52.329163 (+ 1199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:10:52.329526 (+ 363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:10:52.329610 (+ 84us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:10:52.331230 (+ 1620us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:10:52.331234 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:10:52.331236 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:10:52.331259 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:10:52.332679 (+ 1420us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:10:52.333158 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:10:52.333161 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:10:52.333163 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:10:52.333238 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:10:52.333558 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:10:52.333561 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:10:52.333563 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:10:52.333734 (+ 171us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:10:52.333880 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0620 06:10:52.334137 (+ 257us) server_negotiation.cc:299] Negotiation successful
0620 06:10:52.334226 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260620 06:10:52.338626 31997 catalog_manager.cc:1347] Generated new cluster ID: fb78a004831e4dbaba7fe7ffdcb77979
I20260620 06:10:52.338716 31997 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 06:10:52.353529 31997 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 06:10:52.354449 31997 catalog_manager.cc:1504] Loading token signing keys...
I20260620 06:10:52.358321 31997 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 695f889cbadf4375907d65a29b67108a: Generated new TSK 0
I20260620 06:10:52.359051 31997 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 06:10:52.368989 11672 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9HMp43/build/debug/bin/kudu with pid 31910
W20260620 06:10:52.374313 31895 connection.cc:535] client connection to 127.0.0.1:43723 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:43723): Connection reset by peer (error 104)
2026-06-20T06:10:52Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (1409 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781935603120653-11672-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 06:10:52 dist-test-slave-9hm5 krb5kdc[32009](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 06:10:52 dist-test-slave-9hm5 krb5kdc[32009](info): set up 2 sockets
Jun 20 06:10:52 dist-test-slave-9hm5 krb5kdc[32009](info): commencing operation
krb5kdc: starting...
W20260620 06:10:53.407904 11672 mini_kdc.cc:121] Time spent starting KDC: real 1.018s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781935603120653-11672-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781935603120653-11672-0/krb5kdc/test-user.keytab.
Jun 20 06:10:53 dist-test-slave-9hm5 krb5kdc[32009](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935853, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T06:10:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T06:10:53Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781935603120653-11672-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781935603120653-11672-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 06:10:53.524937 11672 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9HMp43/build/debug/bin/kudu
/tmp/dist-test-task9HMp43/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781935603120653-11672-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781935603120653-11672-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781935603120653-11672-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781935603120653-11672-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38185
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781935603120653-11672-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44949
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38185
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781935603120653-11672-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781935603120653-11672-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781935603120653-11672-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 06:10:53.639886 32022 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:10:53.640192 32022 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:10:53.640234 32022 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:10:53.645550 32022 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 06:10:53.645627 32022 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:10:53.645689 32022 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:10:53.645725 32022 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 06:10:53.645756 32022 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 06:10:53.652437 32022 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44949
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781935603120653-11672-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781935603120653-11672-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38185
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781935603120653-11672-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38185
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781935603120653-11672-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32022
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781935603120653-11672-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 05:28:48 UTC on ccycloud.quasar-fixhms.root.comops.site
build id 4457020
I20260620 06:10:53.653533 32022 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:10:53.654560 32022 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 06:10:53.663200 32027 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:10:53.663250 32030 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:10:53.663276 32028 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:10:53.663727 32022 server_base.cc:1034] running on GCE node
I20260620 06:10:53.664296 32022 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:10:53.665400 32022 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:10:53.666561 32022 hybrid_clock.cc:648] HybridClock initialized: now 1781935853666535 us; error 35 us; skew 500 ppm
Jun 20 06:10:53 dist-test-slave-9hm5 krb5kdc[32009](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935853, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 06:10:53.669313 32022 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 06:10:53.670154 32022 webserver.cc:460] Webserver started at http://127.0.0.1:33671/ using document root <none> and password file <none>
I20260620 06:10:53.670712 32022 fs_manager.cc:359] Metadata directory not provided
I20260620 06:10:53.670802 32022 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:10:53.671046 32022 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 06:10:53.673154 32022 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781935603120653-11672-0/minicluster-data/master-0/data/instance:
uuid: "632587c90558449c84a39bfa1404bb84"
format_stamp: "Formatted at 2026-06-20 06:10:53 on dist-test-slave-9hm5"
server_key: "1e853652590548565ecc69ecd368051d"
server_key_iv: "ca2c89c2b67abecf0ac27aa25f6a13d1"
server_key_version: "encryptionkey@0"
I20260620 06:10:53.673730 32022 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781935603120653-11672-0/minicluster-data/master-0/wal/instance:
uuid: "632587c90558449c84a39bfa1404bb84"
format_stamp: "Formatted at 2026-06-20 06:10:53 on dist-test-slave-9hm5"
server_key: "1e853652590548565ecc69ecd368051d"
server_key_iv: "ca2c89c2b67abecf0ac27aa25f6a13d1"
server_key_version: "encryptionkey@0"
I20260620 06:10:53.677561 32022 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260620 06:10:53.679888 32037 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:10:53.681109 32022 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 06:10:53.681249 32022 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781935603120653-11672-0/minicluster-data/master-0/data,/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781935603120653-11672-0/minicluster-data/master-0/wal
uuid: "632587c90558449c84a39bfa1404bb84"
format_stamp: "Formatted at 2026-06-20 06:10:53 on dist-test-slave-9hm5"
server_key: "1e853652590548565ecc69ecd368051d"
server_key_iv: "ca2c89c2b67abecf0ac27aa25f6a13d1"
server_key_version: "encryptionkey@0"
I20260620 06:10:53.681370 32022 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781935603120653-11672-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781935603120653-11672-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781935603120653-11672-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:10:53.695858 32022 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:10:53.698697 32022 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:10:53.698921 32022 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:10:53.706005 32022 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38185
I20260620 06:10:53.706027 32089 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38185 every 8 connection(s)
I20260620 06:10:53.706926 32022 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781935603120653-11672-0/minicluster-data/master-0/data/info.pb
I20260620 06:10:53.709708 32090 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 06:10:53.711102 11672 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9HMp43/build/debug/bin/kudu as pid 32022
I20260620 06:10:53.711246 11672 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781935603120653-11672-0/minicluster-data/master-0/wal/instance
I20260620 06:10:53.711587 11672 external_mini_cluster.cc:1428] Setting key 34af1c78732f627c74e643c6f9422f37
I20260620 06:10:53.715756 32090 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 632587c90558449c84a39bfa1404bb84: Bootstrap starting.
I20260620 06:10:53.717897 32090 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 632587c90558449c84a39bfa1404bb84: Neither blocks nor log segments found. Creating new log.
I20260620 06:10:53.718750 32090 log.cc:824] T 00000000000000000000000000000000 P 632587c90558449c84a39bfa1404bb84: Log is configured to *not* fsync() on all Append() calls
I20260620 06:10:53.720564 32093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:10:53.713045 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53179 (local address 127.0.0.1:38185)
0620 06:10:53.713461 (+ 416us) server_negotiation.cc:206] Beginning negotiation
0620 06:10:53.713476 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 06:10:53.713511 (+ 35us) server_negotiation.cc:407] Connection header received
0620 06:10:53.714126 (+ 615us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:10:53.714142 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:10:53.714460 (+ 318us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:10:53.714799 (+ 339us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:10:53.715177 (+ 378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:10:53.716254 (+ 1077us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:10:53.717130 (+ 876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:10:53.717594 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:10:53.717835 (+ 241us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:10:53.718803 (+ 968us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:10:53.718809 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:10:53.718821 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 06:10:53.719264 (+ 443us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:10:53.719322 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 06:10:53.719356 (+ 34us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:10:53.719498 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0620 06:10:53.719722 (+ 224us) server_negotiation.cc:299] Negotiation successful
0620 06:10:53.719866 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":282,"thread_start_us":128,"threads_started":1}
I20260620 06:10:53.720890 32090 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 632587c90558449c84a39bfa1404bb84: No bootstrap required, opened a new log
I20260620 06:10:53.723481 11672 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 06:10:53.724679 32090 raft_consensus.cc:348] T 00000000000000000000000000000000 P 632587c90558449c84a39bfa1404bb84 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "632587c90558449c84a39bfa1404bb84" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38185 } }
I20260620 06:10:53.724943 32090 raft_consensus.cc:374] T 00000000000000000000000000000000 P 632587c90558449c84a39bfa1404bb84 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 06:10:53.725031 32090 raft_consensus.cc:729] T 00000000000000000000000000000000 P 632587c90558449c84a39bfa1404bb84 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 632587c90558449c84a39bfa1404bb84, State: Initialized, Role: FOLLOWER
I20260620 06:10:53.725530 32090 consensus_queue.cc:260] T 00000000000000000000000000000000 P 632587c90558449c84a39bfa1404bb84 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "632587c90558449c84a39bfa1404bb84" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38185 } }
I20260620 06:10:53.725697 32090 raft_consensus.cc:388] T 00000000000000000000000000000000 P 632587c90558449c84a39bfa1404bb84 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 06:10:53.725781 32090 raft_consensus.cc:482] T 00000000000000000000000000000000 P 632587c90558449c84a39bfa1404bb84 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 06:10:53.725878 32090 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 632587c90558449c84a39bfa1404bb84 [term 0 FOLLOWER]: Advancing to term 1
I20260620 06:10:53.726861 32090 raft_consensus.cc:504] T 00000000000000000000000000000000 P 632587c90558449c84a39bfa1404bb84 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "632587c90558449c84a39bfa1404bb84" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38185 } }
I20260620 06:10:53.727210 32090 leader_election.cc:302] T 00000000000000000000000000000000 P 632587c90558449c84a39bfa1404bb84 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 632587c90558449c84a39bfa1404bb84; no voters:
I20260620 06:10:53.727542 32090 leader_election.cc:288] T 00000000000000000000000000000000 P 632587c90558449c84a39bfa1404bb84 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 06:10:53.727635 32100 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 632587c90558449c84a39bfa1404bb84 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 06:10:53.727861 32100 raft_consensus.cc:686] T 00000000000000000000000000000000 P 632587c90558449c84a39bfa1404bb84 [term 1 LEADER]: Becoming Leader. State: Replica: 632587c90558449c84a39bfa1404bb84, State: Running, Role: LEADER
I20260620 06:10:53.728160 32100 consensus_queue.cc:237] T 00000000000000000000000000000000 P 632587c90558449c84a39bfa1404bb84 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "632587c90558449c84a39bfa1404bb84" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38185 } }
I20260620 06:10:53.728922 32090 sys_catalog.cc:564] T 00000000000000000000000000000000 P 632587c90558449c84a39bfa1404bb84 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 06:10:53.730226 32093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:10:53.725608 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60957 (local address 127.0.0.1:38185)
0620 06:10:53.725742 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0620 06:10:53.725746 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 06:10:53.725972 (+ 226us) server_negotiation.cc:407] Connection header received
0620 06:10:53.726121 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:10:53.726124 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:10:53.726176 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:10:53.726257 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:10:53.726530 (+ 273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:10:53.727125 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:10:53.728248 (+ 1123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:10:53.728544 (+ 296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:10:53.728607 (+ 63us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:10:53.729569 (+ 962us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:10:53.729573 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:10:53.729576 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 06:10:53.729589 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:10:53.729649 (+ 60us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 06:10:53.729667 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:10:53.729753 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0620 06:10:53.729999 (+ 246us) server_negotiation.cc:299] Negotiation successful
0620 06:10:53.730042 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260620 06:10:53.730120 32102 sys_catalog.cc:455] T 00000000000000000000000000000000 P 632587c90558449c84a39bfa1404bb84 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "632587c90558449c84a39bfa1404bb84" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "632587c90558449c84a39bfa1404bb84" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38185 } } }
I20260620 06:10:53.730854 32102 sys_catalog.cc:458] T 00000000000000000000000000000000 P 632587c90558449c84a39bfa1404bb84 [sys.catalog]: This master's current role is: LEADER
I20260620 06:10:53.731992 32113 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 06:10:53.731848 32101 sys_catalog.cc:455] T 00000000000000000000000000000000 P 632587c90558449c84a39bfa1404bb84 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 632587c90558449c84a39bfa1404bb84. Latest consensus state: current_term: 1 leader_uuid: "632587c90558449c84a39bfa1404bb84" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "632587c90558449c84a39bfa1404bb84" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38185 } } }
I20260620 06:10:53.732599 32101 sys_catalog.cc:458] T 00000000000000000000000000000000 P 632587c90558449c84a39bfa1404bb84 [sys.catalog]: This master's current role is: LEADER
I20260620 06:10:53.735160 32113 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 06:10:53.740684 32113 catalog_manager.cc:1347] Generated new cluster ID: de5c802155e940dfbcdf2e6ff603057a
I20260620 06:10:53.740756 32113 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 06:10:53.759382 32113 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 06:10:53.760509 32113 catalog_manager.cc:1504] Loading token signing keys...
I20260620 06:10:53.769461 32113 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 632587c90558449c84a39bfa1404bb84: Generated new TSK 0
I20260620 06:10:53.770253 32113 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 06:10:53.774116 11672 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9HMp43/build/debug/bin/kudu with pid 32022
W20260620 06:10:53.779294 32007 connection.cc:535] client connection to 127.0.0.1:38185 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:38185): Connection reset by peer (error 104)
2026-06-20T06:10:53Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1404 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781935603120653-11672-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 06:10:53 dist-test-slave-9hm5 krb5kdc[32121](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 06:10:53 dist-test-slave-9hm5 krb5kdc[32121](info): set up 2 sockets
Jun 20 06:10:53 dist-test-slave-9hm5 krb5kdc[32121](info): commencing operation
krb5kdc: starting...
W20260620 06:10:54.811877 11672 mini_kdc.cc:121] Time spent starting KDC: real 1.018s user 0.006s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781935603120653-11672-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781935603120653-11672-0/krb5kdc/test-user.keytab.
Jun 20 06:10:54 dist-test-slave-9hm5 krb5kdc[32121](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935854, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T06:10:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T06:10:54Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781935603120653-11672-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781935603120653-11672-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 06:10:54.930591 11672 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9HMp43/build/debug/bin/kudu
/tmp/dist-test-task9HMp43/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781935603120653-11672-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781935603120653-11672-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781935603120653-11672-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781935603120653-11672-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34797
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781935603120653-11672-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36587
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34797
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781935603120653-11672-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781935603120653-11672-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781935603120653-11672-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 06:10:55.040422 32134 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:10:55.040773 32134 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:10:55.040869 32134 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:10:55.044972 32134 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 06:10:55.045083 32134 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:10:55.045126 32134 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:10:55.045163 32134 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 06:10:55.045195 32134 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 06:10:55.050583 32134 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36587
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781935603120653-11672-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781935603120653-11672-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34797
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781935603120653-11672-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34797
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781935603120653-11672-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32134
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781935603120653-11672-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 05:28:48 UTC on ccycloud.quasar-fixhms.root.comops.site
build id 4457020
I20260620 06:10:55.051698 32134 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:10:55.052738 32134 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 06:10:55.061300 32140 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:10:55.061492 32134 server_base.cc:1034] running on GCE node
W20260620 06:10:55.061311 32142 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:10:55.061324 32139 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:10:55.062279 32134 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:10:55.063445 32134 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:10:55.064636 32134 hybrid_clock.cc:648] HybridClock initialized: now 1781935855064618 us; error 42 us; skew 500 ppm
Jun 20 06:10:55 dist-test-slave-9hm5 krb5kdc[32121](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935855, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 06:10:55.067902 32134 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 06:10:55.068910 32134 webserver.cc:460] Webserver started at http://127.0.0.1:36015/ using document root <none> and password file <none>
I20260620 06:10:55.069499 32134 fs_manager.cc:359] Metadata directory not provided
I20260620 06:10:55.069595 32134 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:10:55.069828 32134 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 06:10:55.072036 32134 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781935603120653-11672-0/minicluster-data/master-0/data/instance:
uuid: "3dc3fc3757b640b7ba9b7d400b926e51"
format_stamp: "Formatted at 2026-06-20 06:10:55 on dist-test-slave-9hm5"
server_key: "1ce9ddb8a0ebab708b3780f77c944c50"
server_key_iv: "006f3c06b5b92a03c4e736a439723f2d"
server_key_version: "encryptionkey@0"
I20260620 06:10:55.072626 32134 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781935603120653-11672-0/minicluster-data/master-0/wal/instance:
uuid: "3dc3fc3757b640b7ba9b7d400b926e51"
format_stamp: "Formatted at 2026-06-20 06:10:55 on dist-test-slave-9hm5"
server_key: "1ce9ddb8a0ebab708b3780f77c944c50"
server_key_iv: "006f3c06b5b92a03c4e736a439723f2d"
server_key_version: "encryptionkey@0"
I20260620 06:10:55.076486 32134 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260620 06:10:55.079087 32149 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:10:55.080543 32134 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 06:10:55.080696 32134 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781935603120653-11672-0/minicluster-data/master-0/data,/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781935603120653-11672-0/minicluster-data/master-0/wal
uuid: "3dc3fc3757b640b7ba9b7d400b926e51"
format_stamp: "Formatted at 2026-06-20 06:10:55 on dist-test-slave-9hm5"
server_key: "1ce9ddb8a0ebab708b3780f77c944c50"
server_key_iv: "006f3c06b5b92a03c4e736a439723f2d"
server_key_version: "encryptionkey@0"
I20260620 06:10:55.080833 32134 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781935603120653-11672-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781935603120653-11672-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781935603120653-11672-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:10:55.085542 32134 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:10:55.088901 32134 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:10:55.089104 32134 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:10:55.096203 32134 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34797
I20260620 06:10:55.096213 32201 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34797 every 8 connection(s)
I20260620 06:10:55.097134 32134 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781935603120653-11672-0/minicluster-data/master-0/data/info.pb
I20260620 06:10:55.099910 32202 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 06:10:55.105225 32202 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3dc3fc3757b640b7ba9b7d400b926e51: Bootstrap starting.
I20260620 06:10:55.106958 11672 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9HMp43/build/debug/bin/kudu as pid 32134
I20260620 06:10:55.107097 11672 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781935603120653-11672-0/minicluster-data/master-0/wal/instance
I20260620 06:10:55.107393 11672 external_mini_cluster.cc:1428] Setting key 36c3f7928ac1815aa11daadd56be667a
I20260620 06:10:55.107892 32202 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3dc3fc3757b640b7ba9b7d400b926e51: Neither blocks nor log segments found. Creating new log.
I20260620 06:10:55.108630 32202 log.cc:824] T 00000000000000000000000000000000 P 3dc3fc3757b640b7ba9b7d400b926e51: Log is configured to *not* fsync() on all Append() calls
I20260620 06:10:55.110692 32202 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3dc3fc3757b640b7ba9b7d400b926e51: No bootstrap required, opened a new log
I20260620 06:10:55.112926 32206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:10:55.108978 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43459 (local address 127.0.0.1:34797)
0620 06:10:55.109404 (+ 426us) server_negotiation.cc:206] Beginning negotiation
0620 06:10:55.109418 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 06:10:55.109454 (+ 36us) server_negotiation.cc:407] Connection header received
0620 06:10:55.110088 (+ 634us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:10:55.110105 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:10:55.110387 (+ 282us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:10:55.110674 (+ 287us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:10:55.111567 (+ 893us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:10:55.111572 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:10:55.111585 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 06:10:55.111596 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:10:55.111716 (+ 120us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 06:10:55.111747 (+ 31us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:10:55.111840 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0620 06:10:55.111995 (+ 155us) server_negotiation.cc:299] Negotiation successful
0620 06:10:55.112169 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":259,"thread_start_us":119,"threads_started":1}
I20260620 06:10:55.114705 32202 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3dc3fc3757b640b7ba9b7d400b926e51 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3dc3fc3757b640b7ba9b7d400b926e51" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34797 } }
I20260620 06:10:55.114969 32202 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3dc3fc3757b640b7ba9b7d400b926e51 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 06:10:55.115088 32202 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3dc3fc3757b640b7ba9b7d400b926e51 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3dc3fc3757b640b7ba9b7d400b926e51, State: Initialized, Role: FOLLOWER
I20260620 06:10:55.115581 32202 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3dc3fc3757b640b7ba9b7d400b926e51 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3dc3fc3757b640b7ba9b7d400b926e51" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34797 } }
I20260620 06:10:55.115794 32202 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3dc3fc3757b640b7ba9b7d400b926e51 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 06:10:55.115806 11672 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 06:10:55.115868 32202 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3dc3fc3757b640b7ba9b7d400b926e51 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 06:10:55.115995 32202 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3dc3fc3757b640b7ba9b7d400b926e51 [term 0 FOLLOWER]: Advancing to term 1
I20260620 06:10:55.117012 32202 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3dc3fc3757b640b7ba9b7d400b926e51 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3dc3fc3757b640b7ba9b7d400b926e51" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34797 } }
I20260620 06:10:55.117379 32202 leader_election.cc:302] T 00000000000000000000000000000000 P 3dc3fc3757b640b7ba9b7d400b926e51 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3dc3fc3757b640b7ba9b7d400b926e51; no voters:
I20260620 06:10:55.117748 32202 leader_election.cc:288] T 00000000000000000000000000000000 P 3dc3fc3757b640b7ba9b7d400b926e51 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 06:10:55.117830 32211 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3dc3fc3757b640b7ba9b7d400b926e51 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 06:10:55.118101 32211 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3dc3fc3757b640b7ba9b7d400b926e51 [term 1 LEADER]: Becoming Leader. State: Replica: 3dc3fc3757b640b7ba9b7d400b926e51, State: Running, Role: LEADER
I20260620 06:10:55.118443 32211 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3dc3fc3757b640b7ba9b7d400b926e51 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3dc3fc3757b640b7ba9b7d400b926e51" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34797 } }
I20260620 06:10:55.118996 32202 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3dc3fc3757b640b7ba9b7d400b926e51 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 06:10:55.119647 32211 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3dc3fc3757b640b7ba9b7d400b926e51 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3dc3fc3757b640b7ba9b7d400b926e51. Latest consensus state: current_term: 1 leader_uuid: "3dc3fc3757b640b7ba9b7d400b926e51" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3dc3fc3757b640b7ba9b7d400b926e51" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34797 } } }
I20260620 06:10:55.119824 32211 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3dc3fc3757b640b7ba9b7d400b926e51 [sys.catalog]: This master's current role is: LEADER
I20260620 06:10:55.120141 32214 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3dc3fc3757b640b7ba9b7d400b926e51 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3dc3fc3757b640b7ba9b7d400b926e51" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3dc3fc3757b640b7ba9b7d400b926e51" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34797 } } }
I20260620 06:10:55.120321 32214 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3dc3fc3757b640b7ba9b7d400b926e51 [sys.catalog]: This master's current role is: LEADER
I20260620 06:10:55.120222 32220 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 06:10:55.122151 32206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:10:55.117871 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50373 (local address 127.0.0.1:34797)
0620 06:10:55.117986 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0620 06:10:55.117992 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 06:10:55.120510 (+ 2518us) server_negotiation.cc:407] Connection header received
0620 06:10:55.120695 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:10:55.120698 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:10:55.120736 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:10:55.120787 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:10:55.121705 (+ 918us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:10:55.121712 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:10:55.121717 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 06:10:55.121732 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:10:55.121765 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 06:10:55.121780 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:10:55.121878 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0620 06:10:55.121996 (+ 118us) server_negotiation.cc:299] Negotiation successful
0620 06:10:55.122036 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260620 06:10:55.123912 32220 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 06:10:55.129133 32220 catalog_manager.cc:1347] Generated new cluster ID: 78f9d683ba2b41739b124a721ada4969
I20260620 06:10:55.129196 32220 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 06:10:55.137868 32220 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 06:10:55.138217 32220 catalog_manager.cc:1504] Loading token signing keys...
I20260620 06:10:55.142700 32220 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3dc3fc3757b640b7ba9b7d400b926e51: Generated new TSK 0
I20260620 06:10:55.143410 32220 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 06:10:55.151392 11672 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9HMp43/build/debug/bin/kudu with pid 32134
2026-06-20T06:10:55Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1376 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781935603120653-11672-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 06:10:55 dist-test-slave-9hm5 krb5kdc[32233](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 06:10:55 dist-test-slave-9hm5 krb5kdc[32233](info): set up 2 sockets
Jun 20 06:10:55 dist-test-slave-9hm5 krb5kdc[32233](info): commencing operation
krb5kdc: starting...
W20260620 06:10:56.196039 11672 mini_kdc.cc:121] Time spent starting KDC: real 1.025s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781935603120653-11672-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781935603120653-11672-0/krb5kdc/test-user.keytab.
Jun 20 06:10:56 dist-test-slave-9hm5 krb5kdc[32233](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935856, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T06:10:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T06:10:56Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781935603120653-11672-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781935603120653-11672-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 06:10:56.317225 11672 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9HMp43/build/debug/bin/kudu
/tmp/dist-test-task9HMp43/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781935603120653-11672-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781935603120653-11672-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781935603120653-11672-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781935603120653-11672-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42151
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781935603120653-11672-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38701
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42151
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781935603120653-11672-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781935603120653-11672-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781935603120653-11672-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 06:10:56.429339 32246 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:10:56.429629 32246 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:10:56.429701 32246 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:10:56.433781 32246 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 06:10:56.433848 32246 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:10:56.433868 32246 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:10:56.433885 32246 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 06:10:56.433944 32246 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 06:10:56.439633 32246 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38701
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781935603120653-11672-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781935603120653-11672-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42151
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781935603120653-11672-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42151
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781935603120653-11672-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32246
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781935603120653-11672-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 05:28:48 UTC on ccycloud.quasar-fixhms.root.comops.site
build id 4457020
I20260620 06:10:56.440789 32246 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:10:56.441859 32246 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 06:10:56.450399 32251 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:10:56.450394 32254 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:10:56.450531 32246 server_base.cc:1034] running on GCE node
W20260620 06:10:56.450773 32252 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:10:56.451372 32246 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:10:56.452462 32246 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:10:56.453631 32246 hybrid_clock.cc:648] HybridClock initialized: now 1781935856453613 us; error 43 us; skew 500 ppm
Jun 20 06:10:56 dist-test-slave-9hm5 krb5kdc[32233](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935856, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 06:10:56.460244 32246 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 06:10:56.461278 32246 webserver.cc:460] Webserver started at http://127.0.0.1:45171/ using document root <none> and password file <none>
I20260620 06:10:56.461880 32246 fs_manager.cc:359] Metadata directory not provided
I20260620 06:10:56.461973 32246 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:10:56.462215 32246 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 06:10:56.464399 32246 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781935603120653-11672-0/minicluster-data/master-0/data/instance:
uuid: "b581c975e10b4a3894120082f7cf0408"
format_stamp: "Formatted at 2026-06-20 06:10:56 on dist-test-slave-9hm5"
server_key: "e5c294f65198a8d15fa2d61731b23744"
server_key_iv: "191df51fb231ab38fd85b261809007c6"
server_key_version: "encryptionkey@0"
I20260620 06:10:56.464972 32246 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781935603120653-11672-0/minicluster-data/master-0/wal/instance:
uuid: "b581c975e10b4a3894120082f7cf0408"
format_stamp: "Formatted at 2026-06-20 06:10:56 on dist-test-slave-9hm5"
server_key: "e5c294f65198a8d15fa2d61731b23744"
server_key_iv: "191df51fb231ab38fd85b261809007c6"
server_key_version: "encryptionkey@0"
I20260620 06:10:56.468701 32246 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.001s
I20260620 06:10:56.471243 32261 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:10:56.472571 32246 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 06:10:56.472714 32246 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781935603120653-11672-0/minicluster-data/master-0/data,/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781935603120653-11672-0/minicluster-data/master-0/wal
uuid: "b581c975e10b4a3894120082f7cf0408"
format_stamp: "Formatted at 2026-06-20 06:10:56 on dist-test-slave-9hm5"
server_key: "e5c294f65198a8d15fa2d61731b23744"
server_key_iv: "191df51fb231ab38fd85b261809007c6"
server_key_version: "encryptionkey@0"
I20260620 06:10:56.472846 32246 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781935603120653-11672-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781935603120653-11672-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781935603120653-11672-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:10:56.505792 32246 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:10:56.509219 32246 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:10:56.509433 32246 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:10:56.516832 32246 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42151
I20260620 06:10:56.516861 32313 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42151 every 8 connection(s)
I20260620 06:10:56.517840 32246 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781935603120653-11672-0/minicluster-data/master-0/data/info.pb
I20260620 06:10:56.520591 32314 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 06:10:56.524036 11672 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9HMp43/build/debug/bin/kudu as pid 32246
I20260620 06:10:56.524214 11672 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781935603120653-11672-0/minicluster-data/master-0/wal/instance
I20260620 06:10:56.524507 11672 external_mini_cluster.cc:1428] Setting key cfe8bedc7bb282fb7588fc3d1b981d6e
I20260620 06:10:56.526490 32314 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b581c975e10b4a3894120082f7cf0408: Bootstrap starting.
I20260620 06:10:56.529000 32314 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b581c975e10b4a3894120082f7cf0408: Neither blocks nor log segments found. Creating new log.
I20260620 06:10:56.529721 32314 log.cc:824] T 00000000000000000000000000000000 P b581c975e10b4a3894120082f7cf0408: Log is configured to *not* fsync() on all Append() calls
I20260620 06:10:56.531853 32314 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b581c975e10b4a3894120082f7cf0408: No bootstrap required, opened a new log
Jun 20 06:10:56 dist-test-slave-9hm5 krb5kdc[32233](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935856, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260620 06:10:56.536517 32314 raft_consensus.cc:348] T 00000000000000000000000000000000 P b581c975e10b4a3894120082f7cf0408 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b581c975e10b4a3894120082f7cf0408" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42151 } }
I20260620 06:10:56.536827 32314 raft_consensus.cc:374] T 00000000000000000000000000000000 P b581c975e10b4a3894120082f7cf0408 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 06:10:56.536890 32314 raft_consensus.cc:729] T 00000000000000000000000000000000 P b581c975e10b4a3894120082f7cf0408 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b581c975e10b4a3894120082f7cf0408, State: Initialized, Role: FOLLOWER
I20260620 06:10:56.537420 32314 consensus_queue.cc:260] T 00000000000000000000000000000000 P b581c975e10b4a3894120082f7cf0408 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b581c975e10b4a3894120082f7cf0408" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42151 } }
I20260620 06:10:56.537586 32314 raft_consensus.cc:388] T 00000000000000000000000000000000 P b581c975e10b4a3894120082f7cf0408 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 06:10:56.537657 32314 raft_consensus.cc:482] T 00000000000000000000000000000000 P b581c975e10b4a3894120082f7cf0408 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 06:10:56.537752 32314 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b581c975e10b4a3894120082f7cf0408 [term 0 FOLLOWER]: Advancing to term 1
I20260620 06:10:56.538915 32314 raft_consensus.cc:504] T 00000000000000000000000000000000 P b581c975e10b4a3894120082f7cf0408 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b581c975e10b4a3894120082f7cf0408" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42151 } }
I20260620 06:10:56.539275 32314 leader_election.cc:302] T 00000000000000000000000000000000 P b581c975e10b4a3894120082f7cf0408 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b581c975e10b4a3894120082f7cf0408; no voters:
I20260620 06:10:56.539669 32314 leader_election.cc:288] T 00000000000000000000000000000000 P b581c975e10b4a3894120082f7cf0408 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 06:10:56.539776 32317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:10:56.525943 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36073 (local address 127.0.0.1:42151)
0620 06:10:56.526388 (+ 445us) server_negotiation.cc:206] Beginning negotiation
0620 06:10:56.526406 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0620 06:10:56.526444 (+ 38us) server_negotiation.cc:407] Connection header received
0620 06:10:56.527125 (+ 681us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:10:56.527143 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:10:56.527500 (+ 357us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:10:56.527873 (+ 373us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:10:56.528764 (+ 891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:10:56.529877 (+ 1113us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:10:56.531003 (+ 1126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:10:56.531464 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:10:56.531588 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:10:56.534430 (+ 2842us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:10:56.534437 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:10:56.534453 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:10:56.534486 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:10:56.536504 (+ 2018us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:10:56.537039 (+ 535us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:10:56.537045 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:10:56.537051 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:10:56.537188 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:10:56.537454 (+ 266us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:10:56.537458 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:10:56.537459 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:10:56.537994 (+ 535us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:10:56.538197 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0620 06:10:56.538434 (+ 237us) server_negotiation.cc:299] Negotiation successful
0620 06:10:56.538743 (+ 309us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":285,"thread_start_us":141,"threads_started":1}
I20260620 06:10:56.539845 32319 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b581c975e10b4a3894120082f7cf0408 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 06:10:56.540083 32319 raft_consensus.cc:686] T 00000000000000000000000000000000 P b581c975e10b4a3894120082f7cf0408 [term 1 LEADER]: Becoming Leader. State: Replica: b581c975e10b4a3894120082f7cf0408, State: Running, Role: LEADER
I20260620 06:10:56.540410 32319 consensus_queue.cc:237] T 00000000000000000000000000000000 P b581c975e10b4a3894120082f7cf0408 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b581c975e10b4a3894120082f7cf0408" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42151 } }
I20260620 06:10:56.540974 32314 sys_catalog.cc:564] T 00000000000000000000000000000000 P b581c975e10b4a3894120082f7cf0408 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 06:10:56.541597 32321 sys_catalog.cc:455] T 00000000000000000000000000000000 P b581c975e10b4a3894120082f7cf0408 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b581c975e10b4a3894120082f7cf0408. Latest consensus state: current_term: 1 leader_uuid: "b581c975e10b4a3894120082f7cf0408" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b581c975e10b4a3894120082f7cf0408" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42151 } } }
I20260620 06:10:56.542618 32321 sys_catalog.cc:458] T 00000000000000000000000000000000 P b581c975e10b4a3894120082f7cf0408 [sys.catalog]: This master's current role is: LEADER
I20260620 06:10:56.542498 32320 sys_catalog.cc:455] T 00000000000000000000000000000000 P b581c975e10b4a3894120082f7cf0408 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b581c975e10b4a3894120082f7cf0408" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b581c975e10b4a3894120082f7cf0408" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42151 } } }
I20260620 06:10:56.542923 11672 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 06:10:56.542933 32320 sys_catalog.cc:458] T 00000000000000000000000000000000 P b581c975e10b4a3894120082f7cf0408 [sys.catalog]: This master's current role is: LEADER
W20260620 06:10:56.546315 32334 catalog_manager.cc:1558] T 00000000000000000000000000000000 P b581c975e10b4a3894120082f7cf0408: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 06:10:56.546407 32334 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 06:10:56.546494 32332 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 06:10:56.547428 32332 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 06:10:56.553341 32332 catalog_manager.cc:1347] Generated new cluster ID: 30e9b575840241c2801f941b128aae55
I20260620 06:10:56.553397 32332 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 06:10:56.555104 32317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:10:56.546542 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48629 (local address 127.0.0.1:42151)
0620 06:10:56.546655 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0620 06:10:56.546659 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 06:10:56.546674 (+ 15us) server_negotiation.cc:407] Connection header received
0620 06:10:56.546710 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:10:56.546713 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:10:56.546754 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:10:56.546851 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:10:56.547705 (+ 854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:10:56.548525 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:10:56.549552 (+ 1027us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:10:56.549783 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:10:56.549897 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:10:56.551668 (+ 1771us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:10:56.551672 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:10:56.551675 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:10:56.551699 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:10:56.553352 (+ 1653us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:10:56.553846 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:10:56.553849 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:10:56.553851 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:10:56.553940 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:10:56.554266 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:10:56.554270 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:10:56.554273 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:10:56.554458 (+ 185us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:10:56.554542 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0620 06:10:56.554855 (+ 313us) server_negotiation.cc:299] Negotiation successful
0620 06:10:56.554963 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260620 06:10:56.559826 32332 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 06:10:56.560953 32332 catalog_manager.cc:1504] Loading token signing keys...
I20260620 06:10:56.581565 32332 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b581c975e10b4a3894120082f7cf0408: Generated new TSK 0
I20260620 06:10:56.582464 32332 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 06:10:56.590035 11672 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9HMp43/build/debug/bin/kudu with pid 32246
2026-06-20T06:10:56Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (1438 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781935603120653-11672-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 06:10:56 dist-test-slave-9hm5 krb5kdc[32345](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 06:10:56 dist-test-slave-9hm5 krb5kdc[32345](info): set up 2 sockets
Jun 20 06:10:56 dist-test-slave-9hm5 krb5kdc[32345](info): commencing operation
krb5kdc: starting...
W20260620 06:10:57.619850 11672 mini_kdc.cc:121] Time spent starting KDC: real 1.010s user 0.001s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781935603120653-11672-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781935603120653-11672-0/krb5kdc/test-user.keytab.
Jun 20 06:10:57 dist-test-slave-9hm5 krb5kdc[32345](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935857, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T06:10:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T06:10:57Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781935603120653-11672-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781935603120653-11672-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 06:10:57.740790 11672 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9HMp43/build/debug/bin/kudu
/tmp/dist-test-task9HMp43/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781935603120653-11672-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781935603120653-11672-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781935603120653-11672-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781935603120653-11672-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38847
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781935603120653-11672-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:46629
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38847
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781935603120653-11672-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781935603120653-11672-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781935603120653-11672-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 06:10:57.852346 32358 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:10:57.852648 32358 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:10:57.852730 32358 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:10:57.856835 32358 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 06:10:57.856916 32358 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:10:57.856954 32358 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:10:57.857010 32358 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 06:10:57.857050 32358 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 06:10:57.862267 32358 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:46629
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781935603120653-11672-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781935603120653-11672-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38847
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781935603120653-11672-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38847
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781935603120653-11672-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32358
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781935603120653-11672-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 05:28:48 UTC on ccycloud.quasar-fixhms.root.comops.site
build id 4457020
I20260620 06:10:57.863281 32358 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:10:57.864311 32358 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 06:10:57.872511 32366 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:10:57.872519 32364 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:10:57.872539 32363 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:10:57.872750 32358 server_base.cc:1034] running on GCE node
I20260620 06:10:57.873370 32358 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:10:57.874425 32358 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:10:57.875674 32358 hybrid_clock.cc:648] HybridClock initialized: now 1781935857875653 us; error 30 us; skew 500 ppm
Jun 20 06:10:57 dist-test-slave-9hm5 krb5kdc[32345](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935857, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 06:10:57.878448 32358 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 06:10:57.879307 32358 webserver.cc:460] Webserver started at http://127.0.0.1:40989/ using document root <none> and password file <none>
I20260620 06:10:57.879896 32358 fs_manager.cc:359] Metadata directory not provided
I20260620 06:10:57.879961 32358 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:10:57.880121 32358 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 06:10:57.882030 32358 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781935603120653-11672-0/minicluster-data/master-0/data/instance:
uuid: "4b404775d9aa4c558d19bf31e82ac108"
format_stamp: "Formatted at 2026-06-20 06:10:57 on dist-test-slave-9hm5"
server_key: "131cb7706bbcdcee75574363c22fbdf2"
server_key_iv: "e0ce5e1826538d38856d3924195eb453"
server_key_version: "encryptionkey@0"
I20260620 06:10:57.882509 32358 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781935603120653-11672-0/minicluster-data/master-0/wal/instance:
uuid: "4b404775d9aa4c558d19bf31e82ac108"
format_stamp: "Formatted at 2026-06-20 06:10:57 on dist-test-slave-9hm5"
server_key: "131cb7706bbcdcee75574363c22fbdf2"
server_key_iv: "e0ce5e1826538d38856d3924195eb453"
server_key_version: "encryptionkey@0"
I20260620 06:10:57.886155 32358 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260620 06:10:57.888664 32373 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:10:57.889946 32358 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 06:10:57.890084 32358 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781935603120653-11672-0/minicluster-data/master-0/data,/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781935603120653-11672-0/minicluster-data/master-0/wal
uuid: "4b404775d9aa4c558d19bf31e82ac108"
format_stamp: "Formatted at 2026-06-20 06:10:57 on dist-test-slave-9hm5"
server_key: "131cb7706bbcdcee75574363c22fbdf2"
server_key_iv: "e0ce5e1826538d38856d3924195eb453"
server_key_version: "encryptionkey@0"
I20260620 06:10:57.890194 32358 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781935603120653-11672-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781935603120653-11672-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781935603120653-11672-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:10:57.900710 32358 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:10:57.903925 32358 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:10:57.904125 32358 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:10:57.911491 32358 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38847
I20260620 06:10:57.911515 32425 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38847 every 8 connection(s)
I20260620 06:10:57.912460 32358 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781935603120653-11672-0/minicluster-data/master-0/data/info.pb
I20260620 06:10:57.915097 32426 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 06:10:57.917027 11672 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9HMp43/build/debug/bin/kudu as pid 32358
I20260620 06:10:57.917171 11672 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781935603120653-11672-0/minicluster-data/master-0/wal/instance
I20260620 06:10:57.917452 11672 external_mini_cluster.cc:1428] Setting key 39369d5a4196f6c45f7d6949e80597d8
I20260620 06:10:57.921530 32426 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4b404775d9aa4c558d19bf31e82ac108: Bootstrap starting.
I20260620 06:10:57.924091 32426 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4b404775d9aa4c558d19bf31e82ac108: Neither blocks nor log segments found. Creating new log.
I20260620 06:10:57.924927 32426 log.cc:824] T 00000000000000000000000000000000 P 4b404775d9aa4c558d19bf31e82ac108: Log is configured to *not* fsync() on all Append() calls
I20260620 06:10:57.926926 32426 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4b404775d9aa4c558d19bf31e82ac108: No bootstrap required, opened a new log
I20260620 06:10:57.927552 32429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:10:57.918876 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37671 (local address 127.0.0.1:38847)
0620 06:10:57.919328 (+ 452us) server_negotiation.cc:206] Beginning negotiation
0620 06:10:57.919348 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0620 06:10:57.919391 (+ 43us) server_negotiation.cc:407] Connection header received
0620 06:10:57.920447 (+ 1056us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:10:57.920471 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:10:57.920884 (+ 413us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:10:57.921227 (+ 343us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:10:57.921629 (+ 402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:10:57.923086 (+ 1457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:10:57.924104 (+ 1018us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:10:57.924795 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:10:57.924960 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:10:57.925776 (+ 816us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:10:57.925781 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:10:57.925797 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 06:10:57.926173 (+ 376us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:10:57.926227 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 06:10:57.926249 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:10:57.926334 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0620 06:10:57.926510 (+ 176us) server_negotiation.cc:299] Negotiation successful
0620 06:10:57.926738 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":299,"thread_start_us":136,"threads_started":1}
I20260620 06:10:57.930400 11672 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 06:10:57.930987 32426 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4b404775d9aa4c558d19bf31e82ac108 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b404775d9aa4c558d19bf31e82ac108" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38847 } }
I20260620 06:10:57.931228 32426 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4b404775d9aa4c558d19bf31e82ac108 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 06:10:57.931305 32426 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4b404775d9aa4c558d19bf31e82ac108 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4b404775d9aa4c558d19bf31e82ac108, State: Initialized, Role: FOLLOWER
I20260620 06:10:57.931798 32426 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4b404775d9aa4c558d19bf31e82ac108 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b404775d9aa4c558d19bf31e82ac108" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38847 } }
I20260620 06:10:57.931955 32426 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4b404775d9aa4c558d19bf31e82ac108 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 06:10:57.932024 32426 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4b404775d9aa4c558d19bf31e82ac108 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 06:10:57.932118 32426 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4b404775d9aa4c558d19bf31e82ac108 [term 0 FOLLOWER]: Advancing to term 1
I20260620 06:10:57.933354 32426 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4b404775d9aa4c558d19bf31e82ac108 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b404775d9aa4c558d19bf31e82ac108" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38847 } }
I20260620 06:10:57.933710 32426 leader_election.cc:302] T 00000000000000000000000000000000 P 4b404775d9aa4c558d19bf31e82ac108 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4b404775d9aa4c558d19bf31e82ac108; no voters:
I20260620 06:10:57.934036 32426 leader_election.cc:288] T 00000000000000000000000000000000 P 4b404775d9aa4c558d19bf31e82ac108 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 06:10:57.934142 32436 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4b404775d9aa4c558d19bf31e82ac108 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 06:10:57.934389 32436 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4b404775d9aa4c558d19bf31e82ac108 [term 1 LEADER]: Becoming Leader. State: Replica: 4b404775d9aa4c558d19bf31e82ac108, State: Running, Role: LEADER
I20260620 06:10:57.934733 32436 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4b404775d9aa4c558d19bf31e82ac108 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b404775d9aa4c558d19bf31e82ac108" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38847 } }
I20260620 06:10:57.935326 32426 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4b404775d9aa4c558d19bf31e82ac108 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 06:10:57.936450 32438 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4b404775d9aa4c558d19bf31e82ac108 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4b404775d9aa4c558d19bf31e82ac108. Latest consensus state: current_term: 1 leader_uuid: "4b404775d9aa4c558d19bf31e82ac108" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b404775d9aa4c558d19bf31e82ac108" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38847 } } }
I20260620 06:10:57.936570 32438 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4b404775d9aa4c558d19bf31e82ac108 [sys.catalog]: This master's current role is: LEADER
I20260620 06:10:57.936709 32437 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4b404775d9aa4c558d19bf31e82ac108 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4b404775d9aa4c558d19bf31e82ac108" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b404775d9aa4c558d19bf31e82ac108" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38847 } } }
I20260620 06:10:57.936813 32437 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4b404775d9aa4c558d19bf31e82ac108 [sys.catalog]: This master's current role is: LEADER
I20260620 06:10:57.937070 32444 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 06:10:57.937592 32429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:10:57.932474 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50865 (local address 127.0.0.1:38847)
0620 06:10:57.932609 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0620 06:10:57.932614 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 06:10:57.932836 (+ 222us) server_negotiation.cc:407] Connection header received
0620 06:10:57.932921 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:10:57.932924 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:10:57.932976 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:10:57.933061 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:10:57.933291 (+ 230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:10:57.934053 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:10:57.935144 (+ 1091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:10:57.935461 (+ 317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:10:57.935585 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:10:57.936467 (+ 882us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:10:57.936471 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:10:57.936475 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 06:10:57.936490 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:10:57.936523 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 06:10:57.936540 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:10:57.936613 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0620 06:10:57.937357 (+ 744us) server_negotiation.cc:299] Negotiation successful
0620 06:10:57.937419 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260620 06:10:57.940531 32444 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 06:10:57.945581 32444 catalog_manager.cc:1347] Generated new cluster ID: a4646f04a67d44d2baeb9b1dd9f39fcc
I20260620 06:10:57.945647 32444 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 06:10:57.958483 32444 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 06:10:57.959474 32444 catalog_manager.cc:1504] Loading token signing keys...
I20260620 06:10:57.967238 32444 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4b404775d9aa4c558d19bf31e82ac108: Generated new TSK 0
I20260620 06:10:57.967953 32444 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 06:10:57.982079 11672 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9HMp43/build/debug/bin/kudu with pid 32358
2026-06-20T06:10:57Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1392 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781935603120653-11672-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 06:10:58 dist-test-slave-9hm5 krb5kdc[32457](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 06:10:58 dist-test-slave-9hm5 krb5kdc[32457](info): set up 2 sockets
Jun 20 06:10:58 dist-test-slave-9hm5 krb5kdc[32457](info): commencing operation
krb5kdc: starting...
W20260620 06:10:59.027918 11672 mini_kdc.cc:121] Time spent starting KDC: real 1.026s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781935603120653-11672-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781935603120653-11672-0/krb5kdc/test-user.keytab.
Jun 20 06:10:59 dist-test-slave-9hm5 krb5kdc[32457](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935859, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T06:10:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T06:10:59Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781935603120653-11672-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781935603120653-11672-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 06:10:59.144716 11672 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9HMp43/build/debug/bin/kudu
/tmp/dist-test-task9HMp43/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781935603120653-11672-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781935603120653-11672-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781935603120653-11672-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781935603120653-11672-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43703
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781935603120653-11672-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35475
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43703
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781935603120653-11672-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781935603120653-11672-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781935603120653-11672-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 06:10:59.252480 32470 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:10:59.252750 32470 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:10:59.252803 32470 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:10:59.256731 32470 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 06:10:59.256789 32470 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:10:59.256809 32470 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:10:59.256826 32470 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 06:10:59.256841 32470 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 06:10:59.262032 32470 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35475
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781935603120653-11672-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781935603120653-11672-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43703
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781935603120653-11672-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43703
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781935603120653-11672-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32470
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781935603120653-11672-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 05:28:48 UTC on ccycloud.quasar-fixhms.root.comops.site
build id 4457020
I20260620 06:10:59.263057 32470 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:10:59.264058 32470 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 06:10:59.272691 32476 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:10:59.272699 32478 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:10:59.272706 32475 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:10:59.273005 32470 server_base.cc:1034] running on GCE node
I20260620 06:10:59.273656 32470 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:10:59.274807 32470 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:10:59.276000 32470 hybrid_clock.cc:648] HybridClock initialized: now 1781935859275986 us; error 27 us; skew 500 ppm
Jun 20 06:10:59 dist-test-slave-9hm5 krb5kdc[32457](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935859, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 06:10:59.278636 32470 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 06:10:59.279560 32470 webserver.cc:460] Webserver started at http://127.0.0.1:36151/ using document root <none> and password file <none>
I20260620 06:10:59.280160 32470 fs_manager.cc:359] Metadata directory not provided
I20260620 06:10:59.280254 32470 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:10:59.280495 32470 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 06:10:59.282490 32470 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781935603120653-11672-0/minicluster-data/master-0/data/instance:
uuid: "e8b4c61cf7264041801743688ff724c6"
format_stamp: "Formatted at 2026-06-20 06:10:59 on dist-test-slave-9hm5"
server_key: "d96accaf789c5167f38c8e2820c1150a"
server_key_iv: "814ed6ff3189fc98fe7580482dfd387b"
server_key_version: "encryptionkey@0"
I20260620 06:10:59.283049 32470 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781935603120653-11672-0/minicluster-data/master-0/wal/instance:
uuid: "e8b4c61cf7264041801743688ff724c6"
format_stamp: "Formatted at 2026-06-20 06:10:59 on dist-test-slave-9hm5"
server_key: "d96accaf789c5167f38c8e2820c1150a"
server_key_iv: "814ed6ff3189fc98fe7580482dfd387b"
server_key_version: "encryptionkey@0"
I20260620 06:10:59.286756 32470 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260620 06:10:59.289417 32485 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:10:59.290578 32470 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260620 06:10:59.290719 32470 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781935603120653-11672-0/minicluster-data/master-0/data,/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781935603120653-11672-0/minicluster-data/master-0/wal
uuid: "e8b4c61cf7264041801743688ff724c6"
format_stamp: "Formatted at 2026-06-20 06:10:59 on dist-test-slave-9hm5"
server_key: "d96accaf789c5167f38c8e2820c1150a"
server_key_iv: "814ed6ff3189fc98fe7580482dfd387b"
server_key_version: "encryptionkey@0"
I20260620 06:10:59.290833 32470 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781935603120653-11672-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781935603120653-11672-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781935603120653-11672-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:10:59.295324 32470 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:10:59.298157 32470 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:10:59.298357 32470 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:10:59.305333 32470 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43703
I20260620 06:10:59.305357 32537 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43703 every 8 connection(s)
I20260620 06:10:59.306243 32470 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781935603120653-11672-0/minicluster-data/master-0/data/info.pb
I20260620 06:10:59.308876 32538 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 06:10:59.310500 11672 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9HMp43/build/debug/bin/kudu as pid 32470
I20260620 06:10:59.310642 11672 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781935603120653-11672-0/minicluster-data/master-0/wal/instance
I20260620 06:10:59.310917 11672 external_mini_cluster.cc:1428] Setting key f340e68552b67b4dd9a6a4020aeb3f20
I20260620 06:10:59.315171 32538 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e8b4c61cf7264041801743688ff724c6: Bootstrap starting.
I20260620 06:10:59.316260 32541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:10:59.312337 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44651 (local address 127.0.0.1:43703)
0620 06:10:59.312792 (+ 455us) server_negotiation.cc:206] Beginning negotiation
0620 06:10:59.312807 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 06:10:59.312867 (+ 60us) server_negotiation.cc:407] Connection header received
0620 06:10:59.313519 (+ 652us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:10:59.313535 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:10:59.313813 (+ 278us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:10:59.314158 (+ 345us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:10:59.314966 (+ 808us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:10:59.314971 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:10:59.314982 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 06:10:59.314993 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:10:59.315046 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 06:10:59.315065 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:10:59.315221 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0620 06:10:59.315289 (+ 68us) server_negotiation.cc:299] Negotiation successful
0620 06:10:59.315455 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":304,"thread_start_us":139,"threads_started":1}
I20260620 06:10:59.317816 32538 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e8b4c61cf7264041801743688ff724c6: Neither blocks nor log segments found. Creating new log.
I20260620 06:10:59.318701 32538 log.cc:824] T 00000000000000000000000000000000 P e8b4c61cf7264041801743688ff724c6: Log is configured to *not* fsync() on all Append() calls
I20260620 06:10:59.319324 11672 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 06:10:59.321175 32538 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e8b4c61cf7264041801743688ff724c6: No bootstrap required, opened a new log
I20260620 06:10:59.323359 32541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:10:59.321628 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46031 (local address 127.0.0.1:43703)
0620 06:10:59.321760 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0620 06:10:59.321764 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 06:10:59.321990 (+ 226us) server_negotiation.cc:407] Connection header received
0620 06:10:59.322114 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:10:59.322116 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:10:59.322145 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:10:59.322207 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:10:59.322934 (+ 727us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:10:59.322937 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:10:59.322939 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 06:10:59.322950 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:10:59.322973 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 06:10:59.322984 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:10:59.323061 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0620 06:10:59.323157 (+ 96us) server_negotiation.cc:299] Negotiation successful
0620 06:10:59.323190 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260620 06:10:59.325567 32538 raft_consensus.cc:348] T 00000000000000000000000000000000 P e8b4c61cf7264041801743688ff724c6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e8b4c61cf7264041801743688ff724c6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43703 } }
I20260620 06:10:59.325790 32538 raft_consensus.cc:374] T 00000000000000000000000000000000 P e8b4c61cf7264041801743688ff724c6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 06:10:59.325827 32538 raft_consensus.cc:729] T 00000000000000000000000000000000 P e8b4c61cf7264041801743688ff724c6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e8b4c61cf7264041801743688ff724c6, State: Initialized, Role: FOLLOWER
I20260620 06:10:59.326241 32538 consensus_queue.cc:260] T 00000000000000000000000000000000 P e8b4c61cf7264041801743688ff724c6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e8b4c61cf7264041801743688ff724c6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43703 } }
I20260620 06:10:59.326371 32538 raft_consensus.cc:388] T 00000000000000000000000000000000 P e8b4c61cf7264041801743688ff724c6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 06:10:59.326418 32538 raft_consensus.cc:482] T 00000000000000000000000000000000 P e8b4c61cf7264041801743688ff724c6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 06:10:59.326490 32538 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e8b4c61cf7264041801743688ff724c6 [term 0 FOLLOWER]: Advancing to term 1
I20260620 06:10:59.327597 32538 raft_consensus.cc:504] T 00000000000000000000000000000000 P e8b4c61cf7264041801743688ff724c6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e8b4c61cf7264041801743688ff724c6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43703 } }
I20260620 06:10:59.327986 32538 leader_election.cc:302] T 00000000000000000000000000000000 P e8b4c61cf7264041801743688ff724c6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e8b4c61cf7264041801743688ff724c6; no voters:
I20260620 06:10:59.328289 32538 leader_election.cc:288] T 00000000000000000000000000000000 P e8b4c61cf7264041801743688ff724c6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 06:10:59.328421 32548 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e8b4c61cf7264041801743688ff724c6 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 06:10:59.328657 32548 raft_consensus.cc:686] T 00000000000000000000000000000000 P e8b4c61cf7264041801743688ff724c6 [term 1 LEADER]: Becoming Leader. State: Replica: e8b4c61cf7264041801743688ff724c6, State: Running, Role: LEADER
I20260620 06:10:59.329041 32548 consensus_queue.cc:237] T 00000000000000000000000000000000 P e8b4c61cf7264041801743688ff724c6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e8b4c61cf7264041801743688ff724c6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43703 } }
I20260620 06:10:59.329519 32538 sys_catalog.cc:564] T 00000000000000000000000000000000 P e8b4c61cf7264041801743688ff724c6 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 06:10:59.330672 32549 sys_catalog.cc:455] T 00000000000000000000000000000000 P e8b4c61cf7264041801743688ff724c6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e8b4c61cf7264041801743688ff724c6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e8b4c61cf7264041801743688ff724c6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43703 } } }
I20260620 06:10:59.330811 32549 sys_catalog.cc:458] T 00000000000000000000000000000000 P e8b4c61cf7264041801743688ff724c6 [sys.catalog]: This master's current role is: LEADER
I20260620 06:10:59.331216 32550 sys_catalog.cc:455] T 00000000000000000000000000000000 P e8b4c61cf7264041801743688ff724c6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e8b4c61cf7264041801743688ff724c6. Latest consensus state: current_term: 1 leader_uuid: "e8b4c61cf7264041801743688ff724c6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e8b4c61cf7264041801743688ff724c6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43703 } } }
I20260620 06:10:59.331450 32550 sys_catalog.cc:458] T 00000000000000000000000000000000 P e8b4c61cf7264041801743688ff724c6 [sys.catalog]: This master's current role is: LEADER
I20260620 06:10:59.331212 32557 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 06:10:59.334317 32557 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 06:10:59.340066 32557 catalog_manager.cc:1347] Generated new cluster ID: 3c2ad7e7e10b413fb812ae20c787a733
I20260620 06:10:59.340137 32557 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 06:10:59.351047 32557 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 06:10:59.351370 32557 catalog_manager.cc:1504] Loading token signing keys...
I20260620 06:10:59.371163 32557 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e8b4c61cf7264041801743688ff724c6: Generated new TSK 0
I20260620 06:10:59.372052 32557 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 06:10:59.385198 11672 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9HMp43/build/debug/bin/kudu with pid 32470
2026-06-20T06:10:59Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1402 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781935603120653-11672-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 06:10:59 dist-test-slave-9hm5 krb5kdc[32569](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 06:10:59 dist-test-slave-9hm5 krb5kdc[32569](info): set up 2 sockets
Jun 20 06:10:59 dist-test-slave-9hm5 krb5kdc[32569](info): commencing operation
krb5kdc: starting...
W20260620 06:11:00.427898 11672 mini_kdc.cc:121] Time spent starting KDC: real 1.023s user 0.001s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781935603120653-11672-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781935603120653-11672-0/krb5kdc/test-user.keytab.
Jun 20 06:11:00 dist-test-slave-9hm5 krb5kdc[32569](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935860, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T06:11:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T06:11:00Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781935603120653-11672-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781935603120653-11672-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 06:11:00.546670 11672 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9HMp43/build/debug/bin/kudu
/tmp/dist-test-task9HMp43/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781935603120653-11672-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781935603120653-11672-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781935603120653-11672-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781935603120653-11672-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41009
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781935603120653-11672-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38649
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41009
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781935603120653-11672-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781935603120653-11672-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781935603120653-11672-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 06:11:00.657336 32582 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:11:00.657619 32582 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:11:00.657661 32582 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:11:00.661703 32582 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 06:11:00.661763 32582 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:11:00.661782 32582 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:11:00.661798 32582 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 06:11:00.661813 32582 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 06:11:00.667142 32582 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38649
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781935603120653-11672-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781935603120653-11672-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41009
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781935603120653-11672-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41009
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781935603120653-11672-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32582
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781935603120653-11672-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 05:28:48 UTC on ccycloud.quasar-fixhms.root.comops.site
build id 4457020
I20260620 06:11:00.668274 32582 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:11:00.669327 32582 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 06:11:00.677958 32587 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:11:00.677983 32590 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:11:00.677982 32588 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:11:00.678303 32582 server_base.cc:1034] running on GCE node
I20260620 06:11:00.678917 32582 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:11:00.680110 32582 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:11:00.681306 32582 hybrid_clock.cc:648] HybridClock initialized: now 1781935860681286 us; error 32 us; skew 500 ppm
Jun 20 06:11:00 dist-test-slave-9hm5 krb5kdc[32569](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935860, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 06:11:00.684321 32582 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 06:11:00.685223 32582 webserver.cc:460] Webserver started at http://127.0.0.1:36849/ using document root <none> and password file <none>
I20260620 06:11:00.685801 32582 fs_manager.cc:359] Metadata directory not provided
I20260620 06:11:00.685894 32582 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:11:00.686131 32582 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 06:11:00.688300 32582 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781935603120653-11672-0/minicluster-data/master-0/data/instance:
uuid: "7e53d1fb4030416cbe0c8bed4dceb6c9"
format_stamp: "Formatted at 2026-06-20 06:11:00 on dist-test-slave-9hm5"
server_key: "f442fd5cb2ebf9529325b4eb0e4f6ef4"
server_key_iv: "0af530d9e320440b45136f19cf74b899"
server_key_version: "encryptionkey@0"
I20260620 06:11:00.688874 32582 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781935603120653-11672-0/minicluster-data/master-0/wal/instance:
uuid: "7e53d1fb4030416cbe0c8bed4dceb6c9"
format_stamp: "Formatted at 2026-06-20 06:11:00 on dist-test-slave-9hm5"
server_key: "f442fd5cb2ebf9529325b4eb0e4f6ef4"
server_key_iv: "0af530d9e320440b45136f19cf74b899"
server_key_version: "encryptionkey@0"
I20260620 06:11:00.692786 32582 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.004s
I20260620 06:11:00.695263 32597 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:11:00.696604 32582 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260620 06:11:00.696751 32582 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781935603120653-11672-0/minicluster-data/master-0/data,/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781935603120653-11672-0/minicluster-data/master-0/wal
uuid: "7e53d1fb4030416cbe0c8bed4dceb6c9"
format_stamp: "Formatted at 2026-06-20 06:11:00 on dist-test-slave-9hm5"
server_key: "f442fd5cb2ebf9529325b4eb0e4f6ef4"
server_key_iv: "0af530d9e320440b45136f19cf74b899"
server_key_version: "encryptionkey@0"
I20260620 06:11:00.696868 32582 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781935603120653-11672-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781935603120653-11672-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781935603120653-11672-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:11:00.719241 32582 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:11:00.722435 32582 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:11:00.722657 32582 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:11:00.730015 32582 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41009
I20260620 06:11:00.730036 32649 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41009 every 8 connection(s)
I20260620 06:11:00.731124 32582 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781935603120653-11672-0/minicluster-data/master-0/data/info.pb
I20260620 06:11:00.733992 32650 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 06:11:00.734061 11672 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9HMp43/build/debug/bin/kudu as pid 32582
I20260620 06:11:00.734174 11672 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781935603120653-11672-0/minicluster-data/master-0/wal/instance
I20260620 06:11:00.734517 11672 external_mini_cluster.cc:1428] Setting key de68d77698c1d378b90f9ec1246544de
I20260620 06:11:00.740453 32650 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7e53d1fb4030416cbe0c8bed4dceb6c9: Bootstrap starting.
I20260620 06:11:00.742621 32650 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7e53d1fb4030416cbe0c8bed4dceb6c9: Neither blocks nor log segments found. Creating new log.
I20260620 06:11:00.743487 32650 log.cc:824] T 00000000000000000000000000000000 P 7e53d1fb4030416cbe0c8bed4dceb6c9: Log is configured to *not* fsync() on all Append() calls
Jun 20 06:11:00 dist-test-slave-9hm5 krb5kdc[32569](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935860, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260620 06:11:00.746008 32650 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7e53d1fb4030416cbe0c8bed4dceb6c9: No bootstrap required, opened a new log
I20260620 06:11:00.749752 32653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:11:00.735951 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.194:43435 (local address 127.0.0.1:41009)
0620 06:11:00.736393 (+ 442us) server_negotiation.cc:206] Beginning negotiation
0620 06:11:00.736408 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 06:11:00.736442 (+ 34us) server_negotiation.cc:407] Connection header received
0620 06:11:00.737208 (+ 766us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:11:00.737225 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:11:00.737599 (+ 374us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:11:00.737945 (+ 346us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:11:00.738777 (+ 832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:11:00.739885 (+ 1108us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:11:00.740850 (+ 965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:11:00.741321 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:11:00.741502 (+ 181us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:11:00.744399 (+ 2897us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:11:00.744406 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:11:00.744423 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:11:00.744454 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:11:00.746489 (+ 2035us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:11:00.747064 (+ 575us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:11:00.747070 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:11:00.747075 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:11:00.747207 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:11:00.747518 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:11:00.747521 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:11:00.747523 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:11:00.748055 (+ 532us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:11:00.748294 (+ 239us) server_negotiation.cc:1035] Waiting for connection context
0620 06:11:00.748556 (+ 262us) server_negotiation.cc:299] Negotiation successful
0620 06:11:00.748800 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":296,"thread_start_us":135,"threads_started":1}
I20260620 06:11:00.750669 32650 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7e53d1fb4030416cbe0c8bed4dceb6c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e53d1fb4030416cbe0c8bed4dceb6c9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41009 } }
I20260620 06:11:00.750941 32650 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7e53d1fb4030416cbe0c8bed4dceb6c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 06:11:00.751009 32650 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7e53d1fb4030416cbe0c8bed4dceb6c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7e53d1fb4030416cbe0c8bed4dceb6c9, State: Initialized, Role: FOLLOWER
I20260620 06:11:00.751435 32650 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7e53d1fb4030416cbe0c8bed4dceb6c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e53d1fb4030416cbe0c8bed4dceb6c9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41009 } }
I20260620 06:11:00.751554 32650 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7e53d1fb4030416cbe0c8bed4dceb6c9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 06:11:00.751659 32650 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7e53d1fb4030416cbe0c8bed4dceb6c9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 06:11:00.751768 32650 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7e53d1fb4030416cbe0c8bed4dceb6c9 [term 0 FOLLOWER]: Advancing to term 1
I20260620 06:11:00.752846 32650 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7e53d1fb4030416cbe0c8bed4dceb6c9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e53d1fb4030416cbe0c8bed4dceb6c9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41009 } }
I20260620 06:11:00.752970 11672 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 06:11:00.753224 32650 leader_election.cc:302] T 00000000000000000000000000000000 P 7e53d1fb4030416cbe0c8bed4dceb6c9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7e53d1fb4030416cbe0c8bed4dceb6c9; no voters:
I20260620 06:11:00.753523 32650 leader_election.cc:288] T 00000000000000000000000000000000 P 7e53d1fb4030416cbe0c8bed4dceb6c9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 06:11:00.753638 32655 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7e53d1fb4030416cbe0c8bed4dceb6c9 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 06:11:00.753880 32655 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7e53d1fb4030416cbe0c8bed4dceb6c9 [term 1 LEADER]: Becoming Leader. State: Replica: 7e53d1fb4030416cbe0c8bed4dceb6c9, State: Running, Role: LEADER
I20260620 06:11:00.754251 32655 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7e53d1fb4030416cbe0c8bed4dceb6c9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e53d1fb4030416cbe0c8bed4dceb6c9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41009 } }
I20260620 06:11:00.754786 32650 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7e53d1fb4030416cbe0c8bed4dceb6c9 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 06:11:00.755584 32655 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7e53d1fb4030416cbe0c8bed4dceb6c9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7e53d1fb4030416cbe0c8bed4dceb6c9. Latest consensus state: current_term: 1 leader_uuid: "7e53d1fb4030416cbe0c8bed4dceb6c9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e53d1fb4030416cbe0c8bed4dceb6c9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41009 } } }
I20260620 06:11:00.755792 32655 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7e53d1fb4030416cbe0c8bed4dceb6c9 [sys.catalog]: This master's current role is: LEADER
I20260620 06:11:00.756033 32657 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7e53d1fb4030416cbe0c8bed4dceb6c9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7e53d1fb4030416cbe0c8bed4dceb6c9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e53d1fb4030416cbe0c8bed4dceb6c9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41009 } } }
I20260620 06:11:00.756131 32657 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7e53d1fb4030416cbe0c8bed4dceb6c9 [sys.catalog]: This master's current role is: LEADER
I20260620 06:11:00.756208 32669 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 06:11:00.759408 32669 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 06:11:00.765009 32653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:11:00.755171 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.194:58583 (local address 127.0.0.1:41009)
0620 06:11:00.755269 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0620 06:11:00.755273 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 06:11:00.756395 (+ 1122us) server_negotiation.cc:407] Connection header received
0620 06:11:00.756618 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:11:00.756622 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:11:00.756677 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:11:00.756732 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:11:00.757727 (+ 995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:11:00.758317 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:11:00.759434 (+ 1117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:11:00.759861 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:11:00.759993 (+ 132us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:11:00.761490 (+ 1497us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:11:00.761493 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:11:00.761496 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:11:00.761520 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:11:00.763060 (+ 1540us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:11:00.763605 (+ 545us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:11:00.763640 (+ 35us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:11:00.763641 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:11:00.763725 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:11:00.764102 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:11:00.764106 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:11:00.764107 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:11:00.764315 (+ 208us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:11:00.764457 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0620 06:11:00.764752 (+ 295us) server_negotiation.cc:299] Negotiation successful
0620 06:11:00.764844 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260620 06:11:00.766475 32669 catalog_manager.cc:1347] Generated new cluster ID: cf366f9690814b339433c7f7314d42ab
I20260620 06:11:00.766538 32669 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 06:11:00.774300 32669 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 06:11:00.775511 32669 catalog_manager.cc:1504] Loading token signing keys...
I20260620 06:11:00.780206 32669 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7e53d1fb4030416cbe0c8bed4dceb6c9: Generated new TSK 0
I20260620 06:11:00.781006 32669 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 06:11:00.787130 11672 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9HMp43/build/debug/bin/kudu with pid 32582
W20260620 06:11:00.792140 32567 connection.cc:535] client connection to 127.0.0.1:41009 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:41009): Connection reset by peer (error 104)
2026-06-20T06:11:00Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (1401 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781935603120653-11672-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 06:11:00 dist-test-slave-9hm5 krb5kdc[32681](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 06:11:00 dist-test-slave-9hm5 krb5kdc[32681](info): set up 2 sockets
Jun 20 06:11:00 dist-test-slave-9hm5 krb5kdc[32681](info): commencing operation
krb5kdc: starting...
W20260620 06:11:01.851768 11672 mini_kdc.cc:121] Time spent starting KDC: real 1.045s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781935603120653-11672-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781935603120653-11672-0/krb5kdc/test-user.keytab.
Jun 20 06:11:01 dist-test-slave-9hm5 krb5kdc[32681](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935861, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T06:11:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T06:11:01Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781935603120653-11672-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781935603120653-11672-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 06:11:01.972568 11672 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9HMp43/build/debug/bin/kudu
/tmp/dist-test-task9HMp43/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781935603120653-11672-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781935603120653-11672-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781935603120653-11672-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781935603120653-11672-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33477
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781935603120653-11672-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34719
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:33477
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781935603120653-11672-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781935603120653-11672-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781935603120653-11672-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 06:11:02.087291 32694 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:11:02.087565 32694 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:11:02.087669 32694 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:11:02.091670 32694 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 06:11:02.091774 32694 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:11:02.091833 32694 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:11:02.091871 32694 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 06:11:02.091934 32694 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 06:11:02.097467 32694 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34719
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781935603120653-11672-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781935603120653-11672-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:33477
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781935603120653-11672-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33477
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781935603120653-11672-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32694
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781935603120653-11672-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 05:28:48 UTC on ccycloud.quasar-fixhms.root.comops.site
build id 4457020
I20260620 06:11:02.098627 32694 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:11:02.099762 32694 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 06:11:02.108372 32700 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:11:02.108382 32699 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:11:02.108366 32702 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:11:02.108666 32694 server_base.cc:1034] running on GCE node
I20260620 06:11:02.109349 32694 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:11:02.110474 32694 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:11:02.111678 32694 hybrid_clock.cc:648] HybridClock initialized: now 1781935862111633 us; error 60 us; skew 500 ppm
Jun 20 06:11:02 dist-test-slave-9hm5 krb5kdc[32681](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935862, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 06:11:02.118916 32694 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 06:11:02.120420 32694 webserver.cc:460] Webserver started at http://127.0.0.1:43967/ using document root <none> and password file <none>
I20260620 06:11:02.121023 32694 fs_manager.cc:359] Metadata directory not provided
I20260620 06:11:02.121088 32694 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:11:02.121331 32694 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 06:11:02.123508 32694 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781935603120653-11672-0/minicluster-data/master-0/data/instance:
uuid: "ef8fddb39e024ce7935eb9b1a81097ca"
format_stamp: "Formatted at 2026-06-20 06:11:02 on dist-test-slave-9hm5"
server_key: "c7a7c5d644d8585259ccdbee1e54cbb7"
server_key_iv: "9bcd5d7fd0d83ffdeb012ef09dcddbaf"
server_key_version: "encryptionkey@0"
I20260620 06:11:02.124189 32694 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781935603120653-11672-0/minicluster-data/master-0/wal/instance:
uuid: "ef8fddb39e024ce7935eb9b1a81097ca"
format_stamp: "Formatted at 2026-06-20 06:11:02 on dist-test-slave-9hm5"
server_key: "c7a7c5d644d8585259ccdbee1e54cbb7"
server_key_iv: "9bcd5d7fd0d83ffdeb012ef09dcddbaf"
server_key_version: "encryptionkey@0"
I20260620 06:11:02.128080 32694 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260620 06:11:02.130523 32709 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:11:02.131850 32694 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 06:11:02.131973 32694 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781935603120653-11672-0/minicluster-data/master-0/data,/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781935603120653-11672-0/minicluster-data/master-0/wal
uuid: "ef8fddb39e024ce7935eb9b1a81097ca"
format_stamp: "Formatted at 2026-06-20 06:11:02 on dist-test-slave-9hm5"
server_key: "c7a7c5d644d8585259ccdbee1e54cbb7"
server_key_iv: "9bcd5d7fd0d83ffdeb012ef09dcddbaf"
server_key_version: "encryptionkey@0"
I20260620 06:11:02.132074 32694 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781935603120653-11672-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781935603120653-11672-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781935603120653-11672-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:11:02.145915 32694 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:11:02.148864 32694 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:11:02.149063 32694 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:11:02.156210 32694 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:33477
I20260620 06:11:02.156230 32761 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:33477 every 8 connection(s)
I20260620 06:11:02.157164 32694 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781935603120653-11672-0/minicluster-data/master-0/data/info.pb
I20260620 06:11:02.158973 11672 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9HMp43/build/debug/bin/kudu as pid 32694
I20260620 06:11:02.159121 11672 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781935603120653-11672-0/minicluster-data/master-0/wal/instance
I20260620 06:11:02.159439 11672 external_mini_cluster.cc:1428] Setting key ed8deffc6ef2727873e6f1c4347ee19d
I20260620 06:11:02.160179 32762 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 06:11:02.165938 32762 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ef8fddb39e024ce7935eb9b1a81097ca: Bootstrap starting.
I20260620 06:11:02.168289 32762 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ef8fddb39e024ce7935eb9b1a81097ca: Neither blocks nor log segments found. Creating new log.
I20260620 06:11:02.168628 32765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:11:02.160894 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.194:51279 (local address 127.0.0.1:33477)
0620 06:11:02.161348 (+ 454us) server_negotiation.cc:206] Beginning negotiation
0620 06:11:02.161364 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0620 06:11:02.161413 (+ 49us) server_negotiation.cc:407] Connection header received
0620 06:11:02.162083 (+ 670us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:11:02.162100 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:11:02.162445 (+ 345us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:11:02.162802 (+ 357us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:11:02.163218 (+ 416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:11:02.164262 (+ 1044us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:11:02.165170 (+ 908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:11:02.165641 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:11:02.165810 (+ 169us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:11:02.166779 (+ 969us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:11:02.166785 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:11:02.166797 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 06:11:02.167194 (+ 397us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:11:02.167257 (+ 63us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 06:11:02.167283 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:11:02.167451 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0620 06:11:02.167681 (+ 230us) server_negotiation.cc:299] Negotiation successful
0620 06:11:02.167834 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":302,"thread_start_us":137,"threads_started":1}
I20260620 06:11:02.169149 32762 log.cc:824] T 00000000000000000000000000000000 P ef8fddb39e024ce7935eb9b1a81097ca: Log is configured to *not* fsync() on all Append() calls
I20260620 06:11:02.171279 32762 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ef8fddb39e024ce7935eb9b1a81097ca: No bootstrap required, opened a new log
I20260620 06:11:02.171595 11672 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 06:11:02.175447 32762 raft_consensus.cc:348] T 00000000000000000000000000000000 P ef8fddb39e024ce7935eb9b1a81097ca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef8fddb39e024ce7935eb9b1a81097ca" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33477 } }
I20260620 06:11:02.175747 32762 raft_consensus.cc:374] T 00000000000000000000000000000000 P ef8fddb39e024ce7935eb9b1a81097ca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 06:11:02.175810 32762 raft_consensus.cc:729] T 00000000000000000000000000000000 P ef8fddb39e024ce7935eb9b1a81097ca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef8fddb39e024ce7935eb9b1a81097ca, State: Initialized, Role: FOLLOWER
I20260620 06:11:02.176263 32762 consensus_queue.cc:260] T 00000000000000000000000000000000 P ef8fddb39e024ce7935eb9b1a81097ca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef8fddb39e024ce7935eb9b1a81097ca" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33477 } }
I20260620 06:11:02.176440 32762 raft_consensus.cc:388] T 00000000000000000000000000000000 P ef8fddb39e024ce7935eb9b1a81097ca [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 06:11:02.176517 32762 raft_consensus.cc:482] T 00000000000000000000000000000000 P ef8fddb39e024ce7935eb9b1a81097ca [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 06:11:02.176648 32762 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ef8fddb39e024ce7935eb9b1a81097ca [term 0 FOLLOWER]: Advancing to term 1
I20260620 06:11:02.177800 32765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:11:02.173617 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.194:52087 (local address 127.0.0.1:33477)
0620 06:11:02.173729 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0620 06:11:02.173735 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 06:11:02.173904 (+ 169us) server_negotiation.cc:407] Connection header received
0620 06:11:02.174018 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:11:02.174020 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:11:02.174055 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:11:02.174118 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:11:02.174463 (+ 345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:11:02.174963 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:11:02.176108 (+ 1145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:11:02.176332 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:11:02.176453 (+ 121us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:11:02.177303 (+ 850us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:11:02.177306 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:11:02.177308 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 06:11:02.177319 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:11:02.177342 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 06:11:02.177352 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:11:02.177416 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0620 06:11:02.177605 (+ 189us) server_negotiation.cc:299] Negotiation successful
0620 06:11:02.177636 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260620 06:11:02.177973 32762 raft_consensus.cc:504] T 00000000000000000000000000000000 P ef8fddb39e024ce7935eb9b1a81097ca [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef8fddb39e024ce7935eb9b1a81097ca" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33477 } }
I20260620 06:11:02.178375 32762 leader_election.cc:302] T 00000000000000000000000000000000 P ef8fddb39e024ce7935eb9b1a81097ca [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ef8fddb39e024ce7935eb9b1a81097ca; no voters:
I20260620 06:11:02.178709 32762 leader_election.cc:288] T 00000000000000000000000000000000 P ef8fddb39e024ce7935eb9b1a81097ca [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 06:11:02.178814 304 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ef8fddb39e024ce7935eb9b1a81097ca [term 1 FOLLOWER]: Leader election won for term 1
I20260620 06:11:02.179044 304 raft_consensus.cc:686] T 00000000000000000000000000000000 P ef8fddb39e024ce7935eb9b1a81097ca [term 1 LEADER]: Becoming Leader. State: Replica: ef8fddb39e024ce7935eb9b1a81097ca, State: Running, Role: LEADER
I20260620 06:11:02.179399 304 consensus_queue.cc:237] T 00000000000000000000000000000000 P ef8fddb39e024ce7935eb9b1a81097ca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef8fddb39e024ce7935eb9b1a81097ca" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33477 } }
I20260620 06:11:02.180071 32762 sys_catalog.cc:564] T 00000000000000000000000000000000 P ef8fddb39e024ce7935eb9b1a81097ca [sys.catalog]: configured and running, proceeding with master startup.
I20260620 06:11:02.181357 305 sys_catalog.cc:455] T 00000000000000000000000000000000 P ef8fddb39e024ce7935eb9b1a81097ca [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ef8fddb39e024ce7935eb9b1a81097ca" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef8fddb39e024ce7935eb9b1a81097ca" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33477 } } }
I20260620 06:11:02.181363 306 sys_catalog.cc:455] T 00000000000000000000000000000000 P ef8fddb39e024ce7935eb9b1a81097ca [sys.catalog]: SysCatalogTable state changed. Reason: New leader ef8fddb39e024ce7935eb9b1a81097ca. Latest consensus state: current_term: 1 leader_uuid: "ef8fddb39e024ce7935eb9b1a81097ca" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef8fddb39e024ce7935eb9b1a81097ca" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33477 } } }
I20260620 06:11:02.181520 306 sys_catalog.cc:458] T 00000000000000000000000000000000 P ef8fddb39e024ce7935eb9b1a81097ca [sys.catalog]: This master's current role is: LEADER
I20260620 06:11:02.181519 305 sys_catalog.cc:458] T 00000000000000000000000000000000 P ef8fddb39e024ce7935eb9b1a81097ca [sys.catalog]: This master's current role is: LEADER
I20260620 06:11:02.182274 313 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 06:11:02.184989 313 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 06:11:02.190336 313 catalog_manager.cc:1347] Generated new cluster ID: 9bca93714e2c4e14841971d425fd83a8
I20260620 06:11:02.190416 313 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 06:11:02.217228 313 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 06:11:02.218593 313 catalog_manager.cc:1504] Loading token signing keys...
I20260620 06:11:02.225257 313 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ef8fddb39e024ce7935eb9b1a81097ca: Generated new TSK 0
I20260620 06:11:02.226130 313 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 06:11:02.232533 11672 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9HMp43/build/debug/bin/kudu with pid 32694
W20260620 06:11:02.237843 32679 connection.cc:535] client connection to 127.0.0.1:33477 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:33477): Connection reset by peer (error 104)
2026-06-20T06:11:02Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1445 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781935603120653-11672-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 06:11:02 dist-test-slave-9hm5 krb5kdc[325](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 06:11:02 dist-test-slave-9hm5 krb5kdc[325](info): set up 2 sockets
Jun 20 06:11:02 dist-test-slave-9hm5 krb5kdc[325](info): commencing operation
krb5kdc: starting...
W20260620 06:11:03.264020 11672 mini_kdc.cc:121] Time spent starting KDC: real 1.012s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781935603120653-11672-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781935603120653-11672-0/krb5kdc/test-user.keytab.
Jun 20 06:11:03 dist-test-slave-9hm5 krb5kdc[325](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935863, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T06:11:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T06:11:03Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781935603120653-11672-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781935603120653-11672-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 06:11:03.386310 11672 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9HMp43/build/debug/bin/kudu
/tmp/dist-test-task9HMp43/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781935603120653-11672-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781935603120653-11672-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781935603120653-11672-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781935603120653-11672-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34363
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781935603120653-11672-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:41797
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34363
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781935603120653-11672-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781935603120653-11672-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781935603120653-11672-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 06:11:03.494669 338 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:11:03.494968 338 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:11:03.495013 338 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:11:03.499372 338 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 06:11:03.499452 338 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:11:03.499472 338 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:11:03.499490 338 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 06:11:03.499505 338 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 06:11:03.504973 338 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:41797
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781935603120653-11672-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781935603120653-11672-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34363
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781935603120653-11672-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34363
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781935603120653-11672-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.338
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781935603120653-11672-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 05:28:48 UTC on ccycloud.quasar-fixhms.root.comops.site
build id 4457020
I20260620 06:11:03.505985 338 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:11:03.506948 338 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 06:11:03.515233 343 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:11:03.515254 344 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:11:03.515334 338 server_base.cc:1034] running on GCE node
W20260620 06:11:03.515251 346 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:11:03.516180 338 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:11:03.517274 338 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:11:03.518435 338 hybrid_clock.cc:648] HybridClock initialized: now 1781935863518419 us; error 31 us; skew 500 ppm
Jun 20 06:11:03 dist-test-slave-9hm5 krb5kdc[325](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935863, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 06:11:03.521219 338 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 06:11:03.522037 338 webserver.cc:460] Webserver started at http://127.0.0.1:33957/ using document root <none> and password file <none>
I20260620 06:11:03.522538 338 fs_manager.cc:359] Metadata directory not provided
I20260620 06:11:03.522603 338 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:11:03.522761 338 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 06:11:03.524783 338 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781935603120653-11672-0/minicluster-data/master-0/data/instance:
uuid: "f059c1c6675547bfae32fb6610218137"
format_stamp: "Formatted at 2026-06-20 06:11:03 on dist-test-slave-9hm5"
server_key: "25b49c523c084689deaee68e8802219d"
server_key_iv: "9471396c9bafc99b6a663a3c547a6091"
server_key_version: "encryptionkey@0"
I20260620 06:11:03.525302 338 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781935603120653-11672-0/minicluster-data/master-0/wal/instance:
uuid: "f059c1c6675547bfae32fb6610218137"
format_stamp: "Formatted at 2026-06-20 06:11:03 on dist-test-slave-9hm5"
server_key: "25b49c523c084689deaee68e8802219d"
server_key_iv: "9471396c9bafc99b6a663a3c547a6091"
server_key_version: "encryptionkey@0"
I20260620 06:11:03.528923 338 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 06:11:03.531164 353 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:11:03.532465 338 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 06:11:03.532616 338 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781935603120653-11672-0/minicluster-data/master-0/data,/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781935603120653-11672-0/minicluster-data/master-0/wal
uuid: "f059c1c6675547bfae32fb6610218137"
format_stamp: "Formatted at 2026-06-20 06:11:03 on dist-test-slave-9hm5"
server_key: "25b49c523c084689deaee68e8802219d"
server_key_iv: "9471396c9bafc99b6a663a3c547a6091"
server_key_version: "encryptionkey@0"
I20260620 06:11:03.532729 338 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781935603120653-11672-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781935603120653-11672-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781935603120653-11672-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:11:03.537199 338 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:11:03.540464 338 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:11:03.540668 338 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:11:03.548384 338 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34363
I20260620 06:11:03.548398 405 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34363 every 8 connection(s)
I20260620 06:11:03.549311 338 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781935603120653-11672-0/minicluster-data/master-0/data/info.pb
I20260620 06:11:03.552100 406 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 06:11:03.552384 11672 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9HMp43/build/debug/bin/kudu as pid 338
I20260620 06:11:03.552513 11672 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781935603120653-11672-0/minicluster-data/master-0/wal/instance
I20260620 06:11:03.552824 11672 external_mini_cluster.cc:1428] Setting key 0f9eb67816226ca3f484cca4a2280bb7
I20260620 06:11:03.557953 409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:11:03.554192 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.194:32885 (local address 127.0.0.1:34363)
0620 06:11:03.554635 (+ 443us) server_negotiation.cc:206] Beginning negotiation
0620 06:11:03.554649 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 06:11:03.554683 (+ 34us) server_negotiation.cc:407] Connection header received
0620 06:11:03.555336 (+ 653us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:11:03.555353 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:11:03.555694 (+ 341us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:11:03.556045 (+ 351us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:11:03.556837 (+ 792us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:11:03.556841 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:11:03.556853 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 06:11:03.556865 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:11:03.556915 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 06:11:03.556934 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:11:03.557080 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0620 06:11:03.557149 (+ 69us) server_negotiation.cc:299] Negotiation successful
0620 06:11:03.557301 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":311,"thread_start_us":142,"threads_started":1}
I20260620 06:11:03.558847 406 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f059c1c6675547bfae32fb6610218137: Bootstrap starting.
I20260620 06:11:03.560820 11672 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 06:11:03.561456 406 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f059c1c6675547bfae32fb6610218137: Neither blocks nor log segments found. Creating new log.
I20260620 06:11:03.562289 406 log.cc:824] T 00000000000000000000000000000000 P f059c1c6675547bfae32fb6610218137: Log is configured to *not* fsync() on all Append() calls
I20260620 06:11:03.564580 406 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f059c1c6675547bfae32fb6610218137: No bootstrap required, opened a new log
I20260620 06:11:03.565255 409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:11:03.562909 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.194:56225 (local address 127.0.0.1:34363)
0620 06:11:03.563046 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0620 06:11:03.563053 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 06:11:03.563854 (+ 801us) server_negotiation.cc:407] Connection header received
0620 06:11:03.563957 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:11:03.563960 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:11:03.563989 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:11:03.564061 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:11:03.564820 (+ 759us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:11:03.564822 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:11:03.564825 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 06:11:03.564836 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:11:03.564858 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 06:11:03.564868 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:11:03.564952 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0620 06:11:03.565074 (+ 122us) server_negotiation.cc:299] Negotiation successful
0620 06:11:03.565108 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260620 06:11:03.568658 406 raft_consensus.cc:348] T 00000000000000000000000000000000 P f059c1c6675547bfae32fb6610218137 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f059c1c6675547bfae32fb6610218137" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34363 } }
I20260620 06:11:03.568915 406 raft_consensus.cc:374] T 00000000000000000000000000000000 P f059c1c6675547bfae32fb6610218137 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 06:11:03.568996 406 raft_consensus.cc:729] T 00000000000000000000000000000000 P f059c1c6675547bfae32fb6610218137 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f059c1c6675547bfae32fb6610218137, State: Initialized, Role: FOLLOWER
I20260620 06:11:03.569399 406 consensus_queue.cc:260] T 00000000000000000000000000000000 P f059c1c6675547bfae32fb6610218137 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f059c1c6675547bfae32fb6610218137" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34363 } }
I20260620 06:11:03.569578 406 raft_consensus.cc:388] T 00000000000000000000000000000000 P f059c1c6675547bfae32fb6610218137 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 06:11:03.569653 406 raft_consensus.cc:482] T 00000000000000000000000000000000 P f059c1c6675547bfae32fb6610218137 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 06:11:03.569765 406 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f059c1c6675547bfae32fb6610218137 [term 0 FOLLOWER]: Advancing to term 1
I20260620 06:11:03.570792 406 raft_consensus.cc:504] T 00000000000000000000000000000000 P f059c1c6675547bfae32fb6610218137 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f059c1c6675547bfae32fb6610218137" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34363 } }
I20260620 06:11:03.571143 406 leader_election.cc:302] T 00000000000000000000000000000000 P f059c1c6675547bfae32fb6610218137 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f059c1c6675547bfae32fb6610218137; no voters:
I20260620 06:11:03.571444 406 leader_election.cc:288] T 00000000000000000000000000000000 P f059c1c6675547bfae32fb6610218137 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 06:11:03.571576 416 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f059c1c6675547bfae32fb6610218137 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 06:11:03.571810 416 raft_consensus.cc:686] T 00000000000000000000000000000000 P f059c1c6675547bfae32fb6610218137 [term 1 LEADER]: Becoming Leader. State: Replica: f059c1c6675547bfae32fb6610218137, State: Running, Role: LEADER
I20260620 06:11:03.572151 416 consensus_queue.cc:237] T 00000000000000000000000000000000 P f059c1c6675547bfae32fb6610218137 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f059c1c6675547bfae32fb6610218137" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34363 } }
I20260620 06:11:03.572705 406 sys_catalog.cc:564] T 00000000000000000000000000000000 P f059c1c6675547bfae32fb6610218137 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 06:11:03.573918 418 sys_catalog.cc:455] T 00000000000000000000000000000000 P f059c1c6675547bfae32fb6610218137 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f059c1c6675547bfae32fb6610218137. Latest consensus state: current_term: 1 leader_uuid: "f059c1c6675547bfae32fb6610218137" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f059c1c6675547bfae32fb6610218137" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34363 } } }
I20260620 06:11:03.573930 417 sys_catalog.cc:455] T 00000000000000000000000000000000 P f059c1c6675547bfae32fb6610218137 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f059c1c6675547bfae32fb6610218137" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f059c1c6675547bfae32fb6610218137" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34363 } } }
I20260620 06:11:03.574110 417 sys_catalog.cc:458] T 00000000000000000000000000000000 P f059c1c6675547bfae32fb6610218137 [sys.catalog]: This master's current role is: LEADER
I20260620 06:11:03.574095 418 sys_catalog.cc:458] T 00000000000000000000000000000000 P f059c1c6675547bfae32fb6610218137 [sys.catalog]: This master's current role is: LEADER
I20260620 06:11:03.574928 425 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 06:11:03.578030 425 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 06:11:03.583196 425 catalog_manager.cc:1347] Generated new cluster ID: ef1d5da78ba94ba7864e18a70615df5a
I20260620 06:11:03.583258 425 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 06:11:03.589380 425 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 06:11:03.589740 425 catalog_manager.cc:1504] Loading token signing keys...
I20260620 06:11:03.600066 425 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f059c1c6675547bfae32fb6610218137: Generated new TSK 0
I20260620 06:11:03.600890 425 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 06:11:03.611577 11672 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9HMp43/build/debug/bin/kudu with pid 338
2026-06-20T06:11:03Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1378 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781935603120653-11672-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 06:11:03 dist-test-slave-9hm5 krb5kdc[437](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 06:11:03 dist-test-slave-9hm5 krb5kdc[437](info): set up 2 sockets
Jun 20 06:11:03 dist-test-slave-9hm5 krb5kdc[437](info): commencing operation
krb5kdc: starting...
W20260620 06:11:04.639950 11672 mini_kdc.cc:121] Time spent starting KDC: real 1.009s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781935603120653-11672-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781935603120653-11672-0/krb5kdc/test-user.keytab.
Jun 20 06:11:04 dist-test-slave-9hm5 krb5kdc[437](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935864, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T06:11:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T06:11:04Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781935603120653-11672-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781935603120653-11672-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 06:11:04.759009 11672 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9HMp43/build/debug/bin/kudu
/tmp/dist-test-task9HMp43/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781935603120653-11672-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781935603120653-11672-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781935603120653-11672-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781935603120653-11672-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33279
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781935603120653-11672-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35453
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:33279
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781935603120653-11672-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781935603120653-11672-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781935603120653-11672-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 06:11:04.866717 450 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:11:04.867063 450 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:11:04.867143 450 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:11:04.871121 450 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 06:11:04.871217 450 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:11:04.871256 450 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:11:04.871312 450 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 06:11:04.871348 450 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 06:11:04.877080 450 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35453
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781935603120653-11672-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781935603120653-11672-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:33279
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781935603120653-11672-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33279
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781935603120653-11672-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.450
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781935603120653-11672-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 05:28:48 UTC on ccycloud.quasar-fixhms.root.comops.site
build id 4457020
I20260620 06:11:04.878129 450 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:11:04.879155 450 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 06:11:04.887694 455 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:11:04.887759 450 server_base.cc:1034] running on GCE node
W20260620 06:11:04.887769 458 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:11:04.887702 456 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:11:04.888518 450 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:11:04.889601 450 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:11:04.890770 450 hybrid_clock.cc:648] HybridClock initialized: now 1781935864890742 us; error 39 us; skew 500 ppm
Jun 20 06:11:04 dist-test-slave-9hm5 krb5kdc[437](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935864, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 06:11:04.893987 450 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 06:11:04.894901 450 webserver.cc:460] Webserver started at http://127.0.0.1:37299/ using document root <none> and password file <none>
I20260620 06:11:04.895423 450 fs_manager.cc:359] Metadata directory not provided
I20260620 06:11:04.895489 450 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:11:04.895737 450 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 06:11:04.897760 450 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781935603120653-11672-0/minicluster-data/master-0/data/instance:
uuid: "9591c3e51fba401d875dda32d4044191"
format_stamp: "Formatted at 2026-06-20 06:11:04 on dist-test-slave-9hm5"
server_key: "401457444ef7ca472faca3611f565303"
server_key_iv: "99896f28b76566046be7ae871ac8859e"
server_key_version: "encryptionkey@0"
I20260620 06:11:04.898296 450 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781935603120653-11672-0/minicluster-data/master-0/wal/instance:
uuid: "9591c3e51fba401d875dda32d4044191"
format_stamp: "Formatted at 2026-06-20 06:11:04 on dist-test-slave-9hm5"
server_key: "401457444ef7ca472faca3611f565303"
server_key_iv: "99896f28b76566046be7ae871ac8859e"
server_key_version: "encryptionkey@0"
I20260620 06:11:04.902175 450 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260620 06:11:04.904676 465 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:11:04.905898 450 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 06:11:04.906029 450 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781935603120653-11672-0/minicluster-data/master-0/data,/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781935603120653-11672-0/minicluster-data/master-0/wal
uuid: "9591c3e51fba401d875dda32d4044191"
format_stamp: "Formatted at 2026-06-20 06:11:04 on dist-test-slave-9hm5"
server_key: "401457444ef7ca472faca3611f565303"
server_key_iv: "99896f28b76566046be7ae871ac8859e"
server_key_version: "encryptionkey@0"
I20260620 06:11:04.906139 450 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781935603120653-11672-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781935603120653-11672-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781935603120653-11672-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:11:04.935811 450 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:11:04.938889 450 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:11:04.939134 450 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:11:04.946219 450 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:33279
I20260620 06:11:04.946246 517 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:33279 every 8 connection(s)
I20260620 06:11:04.947170 450 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781935603120653-11672-0/minicluster-data/master-0/data/info.pb
I20260620 06:11:04.949894 518 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 06:11:04.955143 11672 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9HMp43/build/debug/bin/kudu as pid 450
I20260620 06:11:04.955304 11672 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781935603120653-11672-0/minicluster-data/master-0/wal/instance
I20260620 06:11:04.955389 518 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9591c3e51fba401d875dda32d4044191: Bootstrap starting.
I20260620 06:11:04.955680 11672 external_mini_cluster.cc:1428] Setting key 6a3e7d6e64dde06d0586894b357c7929
I20260620 06:11:04.958070 518 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9591c3e51fba401d875dda32d4044191: Neither blocks nor log segments found. Creating new log.
I20260620 06:11:04.958925 518 log.cc:824] T 00000000000000000000000000000000 P 9591c3e51fba401d875dda32d4044191: Log is configured to *not* fsync() on all Append() calls
I20260620 06:11:04.961112 518 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9591c3e51fba401d875dda32d4044191: No bootstrap required, opened a new log
I20260620 06:11:04.965157 518 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9591c3e51fba401d875dda32d4044191 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9591c3e51fba401d875dda32d4044191" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33279 } }
Jun 20 06:11:04 dist-test-slave-9hm5 krb5kdc[437](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935864, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260620 06:11:04.965431 518 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9591c3e51fba401d875dda32d4044191 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 06:11:04.965499 518 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9591c3e51fba401d875dda32d4044191 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9591c3e51fba401d875dda32d4044191, State: Initialized, Role: FOLLOWER
I20260620 06:11:04.966014 518 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9591c3e51fba401d875dda32d4044191 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9591c3e51fba401d875dda32d4044191" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33279 } }
I20260620 06:11:04.966161 518 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9591c3e51fba401d875dda32d4044191 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 06:11:04.966259 518 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9591c3e51fba401d875dda32d4044191 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 06:11:04.966382 518 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9591c3e51fba401d875dda32d4044191 [term 0 FOLLOWER]: Advancing to term 1
I20260620 06:11:04.967439 518 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9591c3e51fba401d875dda32d4044191 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9591c3e51fba401d875dda32d4044191" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33279 } }
I20260620 06:11:04.967854 518 leader_election.cc:302] T 00000000000000000000000000000000 P 9591c3e51fba401d875dda32d4044191 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9591c3e51fba401d875dda32d4044191; no voters:
I20260620 06:11:04.968182 518 leader_election.cc:288] T 00000000000000000000000000000000 P 9591c3e51fba401d875dda32d4044191 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 06:11:04.968240 523 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9591c3e51fba401d875dda32d4044191 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 06:11:04.968462 523 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9591c3e51fba401d875dda32d4044191 [term 1 LEADER]: Becoming Leader. State: Replica: 9591c3e51fba401d875dda32d4044191, State: Running, Role: LEADER
I20260620 06:11:04.968794 523 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9591c3e51fba401d875dda32d4044191 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9591c3e51fba401d875dda32d4044191" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33279 } }
I20260620 06:11:04.969434 518 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9591c3e51fba401d875dda32d4044191 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 06:11:04.969944 523 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9591c3e51fba401d875dda32d4044191 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9591c3e51fba401d875dda32d4044191" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9591c3e51fba401d875dda32d4044191" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33279 } } }
I20260620 06:11:04.970114 523 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9591c3e51fba401d875dda32d4044191 [sys.catalog]: This master's current role is: LEADER
I20260620 06:11:04.970337 524 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9591c3e51fba401d875dda32d4044191 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9591c3e51fba401d875dda32d4044191. Latest consensus state: current_term: 1 leader_uuid: "9591c3e51fba401d875dda32d4044191" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9591c3e51fba401d875dda32d4044191" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33279 } } }
I20260620 06:11:04.970459 524 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9591c3e51fba401d875dda32d4044191 [sys.catalog]: This master's current role is: LEADER
I20260620 06:11:04.970559 530 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 06:11:04.971738 521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:11:04.957029 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.194:45457 (local address 127.0.0.1:33279)
0620 06:11:04.957439 (+ 410us) server_negotiation.cc:206] Beginning negotiation
0620 06:11:04.957459 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0620 06:11:04.957500 (+ 41us) server_negotiation.cc:407] Connection header received
0620 06:11:04.958357 (+ 857us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:11:04.958380 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:11:04.958802 (+ 422us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:11:04.959223 (+ 421us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:11:04.960139 (+ 916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:11:04.961573 (+ 1434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:11:04.962443 (+ 870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:11:04.963166 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:11:04.963333 (+ 167us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:11:04.966072 (+ 2739us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:11:04.966077 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:11:04.966088 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:11:04.966112 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:11:04.968121 (+ 2009us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:11:04.968717 (+ 596us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:11:04.968722 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:11:04.968728 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:11:04.968857 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:11:04.969218 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:11:04.969222 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:11:04.969223 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:11:04.969692 (+ 469us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:11:04.969944 (+ 252us) server_negotiation.cc:1035] Waiting for connection context
0620 06:11:04.970509 (+ 565us) server_negotiation.cc:299] Negotiation successful
0620 06:11:04.970742 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":267,"thread_start_us":107,"threads_started":1}
I20260620 06:11:04.974757 530 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 06:11:04.975353 11672 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 06:11:04.981844 530 catalog_manager.cc:1347] Generated new cluster ID: 8314c5f2f018491a9abbf0a922bf375d
I20260620 06:11:04.981927 530 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 06:11:04.986080 521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:11:04.977367 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.194:35853 (local address 127.0.0.1:33279)
0620 06:11:04.977470 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0620 06:11:04.977474 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 06:11:04.977744 (+ 270us) server_negotiation.cc:407] Connection header received
0620 06:11:04.977857 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:11:04.977860 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:11:04.977907 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:11:04.977988 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:11:04.978801 (+ 813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:11:04.979699 (+ 898us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:11:04.980533 (+ 834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:11:04.980782 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:11:04.980899 (+ 117us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:11:04.982424 (+ 1525us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:11:04.982428 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:11:04.982430 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:11:04.982454 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:11:04.984133 (+ 1679us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:11:04.984644 (+ 511us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:11:04.984647 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:11:04.984649 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:11:04.984723 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:11:04.985024 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:11:04.985027 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:11:04.985028 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:11:04.985200 (+ 172us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:11:04.985313 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0620 06:11:04.985849 (+ 536us) server_negotiation.cc:299] Negotiation successful
0620 06:11:04.985931 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260620 06:11:05.001098 530 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 06:11:05.002043 530 catalog_manager.cc:1504] Loading token signing keys...
I20260620 06:11:05.013921 530 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9591c3e51fba401d875dda32d4044191: Generated new TSK 0
I20260620 06:11:05.014670 530 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 06:11:05.023061 11672 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9HMp43/build/debug/bin/kudu with pid 450
W20260620 06:11:05.028352 435 connection.cc:535] client connection to 127.0.0.1:33279 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:33279): Connection reset by peer (error 104)
2026-06-20T06:11:05Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (1411 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781935603120653-11672-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 06:11:05 dist-test-slave-9hm5 krb5kdc[549](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 06:11:05 dist-test-slave-9hm5 krb5kdc[549](info): set up 2 sockets
Jun 20 06:11:05 dist-test-slave-9hm5 krb5kdc[549](info): commencing operation
krb5kdc: starting...
W20260620 06:11:06.055893 11672 mini_kdc.cc:121] Time spent starting KDC: real 1.013s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781935603120653-11672-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781935603120653-11672-0/krb5kdc/test-user.keytab.
Jun 20 06:11:06 dist-test-slave-9hm5 krb5kdc[549](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935866, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T06:11:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T06:11:06Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781935603120653-11672-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781935603120653-11672-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 06:11:06.174760 11672 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9HMp43/build/debug/bin/kudu
/tmp/dist-test-task9HMp43/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781935603120653-11672-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781935603120653-11672-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781935603120653-11672-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781935603120653-11672-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33155
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781935603120653-11672-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43305
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:33155
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781935603120653-11672-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781935603120653-11672-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781935603120653-11672-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 06:11:06.282763 562 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:11:06.283093 562 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:11:06.283180 562 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:11:06.287195 562 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 06:11:06.287290 562 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:11:06.287330 562 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:11:06.287392 562 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 06:11:06.287431 562 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 06:11:06.292695 562 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43305
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781935603120653-11672-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781935603120653-11672-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:33155
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781935603120653-11672-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33155
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781935603120653-11672-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.562
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781935603120653-11672-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 05:28:48 UTC on ccycloud.quasar-fixhms.root.comops.site
build id 4457020
I20260620 06:11:06.293774 562 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:11:06.294795 562 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 06:11:06.303337 562 server_base.cc:1034] running on GCE node
W20260620 06:11:06.303292 567 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:11:06.303294 568 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:11:06.303303 570 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:11:06.304277 562 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:11:06.305411 562 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:11:06.306608 562 hybrid_clock.cc:648] HybridClock initialized: now 1781935866306596 us; error 35 us; skew 500 ppm
Jun 20 06:11:06 dist-test-slave-9hm5 krb5kdc[549](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935866, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 06:11:06.309409 562 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 06:11:06.310326 562 webserver.cc:460] Webserver started at http://127.0.0.1:45875/ using document root <none> and password file <none>
I20260620 06:11:06.310837 562 fs_manager.cc:359] Metadata directory not provided
I20260620 06:11:06.310894 562 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:11:06.311066 562 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 06:11:06.313091 562 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781935603120653-11672-0/minicluster-data/master-0/data/instance:
uuid: "60ff7ec02ef641d68334616d51484389"
format_stamp: "Formatted at 2026-06-20 06:11:06 on dist-test-slave-9hm5"
server_key: "5b6828f87661892578c310620b6e1253"
server_key_iv: "e621090c46400555c450a8caf3a79439"
server_key_version: "encryptionkey@0"
I20260620 06:11:06.313616 562 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781935603120653-11672-0/minicluster-data/master-0/wal/instance:
uuid: "60ff7ec02ef641d68334616d51484389"
format_stamp: "Formatted at 2026-06-20 06:11:06 on dist-test-slave-9hm5"
server_key: "5b6828f87661892578c310620b6e1253"
server_key_iv: "e621090c46400555c450a8caf3a79439"
server_key_version: "encryptionkey@0"
I20260620 06:11:06.317243 562 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260620 06:11:06.319574 577 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:11:06.320828 562 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 06:11:06.320925 562 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781935603120653-11672-0/minicluster-data/master-0/data,/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781935603120653-11672-0/minicluster-data/master-0/wal
uuid: "60ff7ec02ef641d68334616d51484389"
format_stamp: "Formatted at 2026-06-20 06:11:06 on dist-test-slave-9hm5"
server_key: "5b6828f87661892578c310620b6e1253"
server_key_iv: "e621090c46400555c450a8caf3a79439"
server_key_version: "encryptionkey@0"
I20260620 06:11:06.321029 562 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781935603120653-11672-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781935603120653-11672-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781935603120653-11672-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:11:06.343406 562 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:11:06.346184 562 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:11:06.346338 562 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:11:06.353396 562 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:33155
I20260620 06:11:06.353420 629 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:33155 every 8 connection(s)
I20260620 06:11:06.354377 562 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781935603120653-11672-0/minicluster-data/master-0/data/info.pb
I20260620 06:11:06.357117 630 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 06:11:06.360566 11672 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9HMp43/build/debug/bin/kudu as pid 562
I20260620 06:11:06.360692 11672 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781935603120653-11672-0/minicluster-data/master-0/wal/instance
I20260620 06:11:06.360963 11672 external_mini_cluster.cc:1428] Setting key 714202d25c4ba30f52e93a4821443879
I20260620 06:11:06.363289 630 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 60ff7ec02ef641d68334616d51484389: Bootstrap starting.
I20260620 06:11:06.366039 630 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 60ff7ec02ef641d68334616d51484389: Neither blocks nor log segments found. Creating new log.
I20260620 06:11:06.366886 630 log.cc:824] T 00000000000000000000000000000000 P 60ff7ec02ef641d68334616d51484389: Log is configured to *not* fsync() on all Append() calls
I20260620 06:11:06.369091 630 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 60ff7ec02ef641d68334616d51484389: No bootstrap required, opened a new log
I20260620 06:11:06.371531 633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:11:06.362401 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.194:45105 (local address 127.0.0.1:33155)
0620 06:11:06.362834 (+ 433us) server_negotiation.cc:206] Beginning negotiation
0620 06:11:06.362853 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0620 06:11:06.362896 (+ 43us) server_negotiation.cc:407] Connection header received
0620 06:11:06.363760 (+ 864us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:11:06.363784 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:11:06.364229 (+ 445us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:11:06.364610 (+ 381us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:11:06.365040 (+ 430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:11:06.366497 (+ 1457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:11:06.367501 (+ 1004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:11:06.368246 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:11:06.368420 (+ 174us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:11:06.369236 (+ 816us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:11:06.369242 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:11:06.369259 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 06:11:06.369774 (+ 515us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:11:06.369840 (+ 66us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 06:11:06.369870 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:11:06.370014 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0620 06:11:06.370178 (+ 164us) server_negotiation.cc:299] Negotiation successful
0620 06:11:06.370317 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":286,"thread_start_us":117,"threads_started":1}
I20260620 06:11:06.373270 630 raft_consensus.cc:348] T 00000000000000000000000000000000 P 60ff7ec02ef641d68334616d51484389 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "60ff7ec02ef641d68334616d51484389" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33155 } }
I20260620 06:11:06.373588 630 raft_consensus.cc:374] T 00000000000000000000000000000000 P 60ff7ec02ef641d68334616d51484389 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 06:11:06.373703 630 raft_consensus.cc:729] T 00000000000000000000000000000000 P 60ff7ec02ef641d68334616d51484389 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60ff7ec02ef641d68334616d51484389, State: Initialized, Role: FOLLOWER
I20260620 06:11:06.374261 630 consensus_queue.cc:260] T 00000000000000000000000000000000 P 60ff7ec02ef641d68334616d51484389 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "60ff7ec02ef641d68334616d51484389" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33155 } }
I20260620 06:11:06.374446 630 raft_consensus.cc:388] T 00000000000000000000000000000000 P 60ff7ec02ef641d68334616d51484389 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 06:11:06.374533 630 raft_consensus.cc:482] T 00000000000000000000000000000000 P 60ff7ec02ef641d68334616d51484389 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 06:11:06.374646 630 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 60ff7ec02ef641d68334616d51484389 [term 0 FOLLOWER]: Advancing to term 1
I20260620 06:11:06.374955 11672 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 06:11:06.375977 630 raft_consensus.cc:504] T 00000000000000000000000000000000 P 60ff7ec02ef641d68334616d51484389 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "60ff7ec02ef641d68334616d51484389" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33155 } }
I20260620 06:11:06.376371 630 leader_election.cc:302] T 00000000000000000000000000000000 P 60ff7ec02ef641d68334616d51484389 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 60ff7ec02ef641d68334616d51484389; no voters:
I20260620 06:11:06.376762 630 leader_election.cc:288] T 00000000000000000000000000000000 P 60ff7ec02ef641d68334616d51484389 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 06:11:06.376860 639 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 60ff7ec02ef641d68334616d51484389 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 06:11:06.377138 639 raft_consensus.cc:686] T 00000000000000000000000000000000 P 60ff7ec02ef641d68334616d51484389 [term 1 LEADER]: Becoming Leader. State: Replica: 60ff7ec02ef641d68334616d51484389, State: Running, Role: LEADER
I20260620 06:11:06.377498 639 consensus_queue.cc:237] T 00000000000000000000000000000000 P 60ff7ec02ef641d68334616d51484389 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "60ff7ec02ef641d68334616d51484389" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33155 } }
I20260620 06:11:06.378149 630 sys_catalog.cc:564] T 00000000000000000000000000000000 P 60ff7ec02ef641d68334616d51484389 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 06:11:06.379272 641 sys_catalog.cc:455] T 00000000000000000000000000000000 P 60ff7ec02ef641d68334616d51484389 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "60ff7ec02ef641d68334616d51484389" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "60ff7ec02ef641d68334616d51484389" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33155 } } }
I20260620 06:11:06.379400 641 sys_catalog.cc:458] T 00000000000000000000000000000000 P 60ff7ec02ef641d68334616d51484389 [sys.catalog]: This master's current role is: LEADER
I20260620 06:11:06.379796 640 sys_catalog.cc:455] T 00000000000000000000000000000000 P 60ff7ec02ef641d68334616d51484389 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 60ff7ec02ef641d68334616d51484389. Latest consensus state: current_term: 1 leader_uuid: "60ff7ec02ef641d68334616d51484389" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "60ff7ec02ef641d68334616d51484389" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33155 } } }
I20260620 06:11:06.379932 640 sys_catalog.cc:458] T 00000000000000000000000000000000 P 60ff7ec02ef641d68334616d51484389 [sys.catalog]: This master's current role is: LEADER
I20260620 06:11:06.380261 649 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 06:11:06.382565 633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:11:06.377607 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.194:40219 (local address 127.0.0.1:33155)
0620 06:11:06.377713 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0620 06:11:06.377718 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 06:11:06.377827 (+ 109us) server_negotiation.cc:407] Connection header received
0620 06:11:06.377995 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:11:06.377999 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:11:06.378051 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:11:06.378137 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:11:06.378578 (+ 441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:11:06.379398 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:11:06.380743 (+ 1345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:11:06.381019 (+ 276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:11:06.381127 (+ 108us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:11:06.382107 (+ 980us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:11:06.382111 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:11:06.382116 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 06:11:06.382132 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:11:06.382156 (+ 24us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 06:11:06.382167 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:11:06.382269 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0620 06:11:06.382389 (+ 120us) server_negotiation.cc:299] Negotiation successful
0620 06:11:06.382423 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260620 06:11:06.384577 649 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 06:11:06.389393 649 catalog_manager.cc:1347] Generated new cluster ID: dc4509527caa4b04ae3210022156f73b
I20260620 06:11:06.389456 649 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 06:11:06.405879 649 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 06:11:06.406806 649 catalog_manager.cc:1504] Loading token signing keys...
I20260620 06:11:06.413157 649 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 60ff7ec02ef641d68334616d51484389: Generated new TSK 0
I20260620 06:11:06.413810 649 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 06:11:06.421566 11672 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9HMp43/build/debug/bin/kudu with pid 562
W20260620 06:11:06.427057 547 connection.cc:535] client connection to 127.0.0.1:33155 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:33155): Connection reset by peer (error 104)
2026-06-20T06:11:06Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1398 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781935603120653-11672-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 06:11:06 dist-test-slave-9hm5 krb5kdc[661](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 06:11:06 dist-test-slave-9hm5 krb5kdc[661](info): set up 2 sockets
Jun 20 06:11:06 dist-test-slave-9hm5 krb5kdc[661](info): commencing operation
krb5kdc: starting...
W20260620 06:11:07.452068 11672 mini_kdc.cc:121] Time spent starting KDC: real 1.011s user 0.004s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781935603120653-11672-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781935603120653-11672-0/krb5kdc/test-user.keytab.
Jun 20 06:11:07 dist-test-slave-9hm5 krb5kdc[661](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935867, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T06:11:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T06:11:07Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781935603120653-11672-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781935603120653-11672-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 06:11:07.570371 11672 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9HMp43/build/debug/bin/kudu
/tmp/dist-test-task9HMp43/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781935603120653-11672-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781935603120653-11672-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781935603120653-11672-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781935603120653-11672-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33729
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781935603120653-11672-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:33633
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:33729
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781935603120653-11672-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781935603120653-11672-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781935603120653-11672-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 06:11:07.684552 674 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:11:07.684887 674 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:11:07.684974 674 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:11:07.689064 674 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 06:11:07.689159 674 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:11:07.689203 674 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:11:07.689265 674 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 06:11:07.689309 674 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 06:11:07.694583 674 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33633
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781935603120653-11672-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781935603120653-11672-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:33729
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781935603120653-11672-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33729
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781935603120653-11672-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.674
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781935603120653-11672-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 05:28:48 UTC on ccycloud.quasar-fixhms.root.comops.site
build id 4457020
I20260620 06:11:07.695693 674 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:11:07.696746 674 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 06:11:07.705304 679 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:11:07.705291 680 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:11:07.705291 682 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:11:07.705582 674 server_base.cc:1034] running on GCE node
I20260620 06:11:07.706151 674 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:11:07.707228 674 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:11:07.708426 674 hybrid_clock.cc:648] HybridClock initialized: now 1781935867708403 us; error 34 us; skew 500 ppm
Jun 20 06:11:07 dist-test-slave-9hm5 krb5kdc[661](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935867, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 06:11:07.714643 674 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 06:11:07.715667 674 webserver.cc:460] Webserver started at http://127.0.0.1:46471/ using document root <none> and password file <none>
I20260620 06:11:07.716248 674 fs_manager.cc:359] Metadata directory not provided
I20260620 06:11:07.716339 674 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:11:07.716580 674 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 06:11:07.718662 674 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781935603120653-11672-0/minicluster-data/master-0/data/instance:
uuid: "443739ddf8f24361bf49f0b5fc726ed5"
format_stamp: "Formatted at 2026-06-20 06:11:07 on dist-test-slave-9hm5"
server_key: "2790ee3146146ad3cc787711f4a7feb0"
server_key_iv: "7de818445a737eb88ed59286572e6968"
server_key_version: "encryptionkey@0"
I20260620 06:11:07.719220 674 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781935603120653-11672-0/minicluster-data/master-0/wal/instance:
uuid: "443739ddf8f24361bf49f0b5fc726ed5"
format_stamp: "Formatted at 2026-06-20 06:11:07 on dist-test-slave-9hm5"
server_key: "2790ee3146146ad3cc787711f4a7feb0"
server_key_iv: "7de818445a737eb88ed59286572e6968"
server_key_version: "encryptionkey@0"
I20260620 06:11:07.722925 674 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260620 06:11:07.725406 689 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:11:07.726802 674 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260620 06:11:07.726969 674 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781935603120653-11672-0/minicluster-data/master-0/data,/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781935603120653-11672-0/minicluster-data/master-0/wal
uuid: "443739ddf8f24361bf49f0b5fc726ed5"
format_stamp: "Formatted at 2026-06-20 06:11:07 on dist-test-slave-9hm5"
server_key: "2790ee3146146ad3cc787711f4a7feb0"
server_key_iv: "7de818445a737eb88ed59286572e6968"
server_key_version: "encryptionkey@0"
I20260620 06:11:07.727097 674 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781935603120653-11672-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781935603120653-11672-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781935603120653-11672-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:11:07.731716 674 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:11:07.734519 674 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:11:07.734709 674 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:11:07.741703 741 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:33729 every 8 connection(s)
I20260620 06:11:07.741700 674 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:33729
I20260620 06:11:07.742640 674 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781935603120653-11672-0/minicluster-data/master-0/data/info.pb
I20260620 06:11:07.745365 742 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 06:11:07.746320 11672 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9HMp43/build/debug/bin/kudu as pid 674
I20260620 06:11:07.746466 11672 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781935603120653-11672-0/minicluster-data/master-0/wal/instance
I20260620 06:11:07.746780 11672 external_mini_cluster.cc:1428] Setting key 0dbac41b6c3e40f9e6525d3bde8dd49a
I20260620 06:11:07.751574 742 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 443739ddf8f24361bf49f0b5fc726ed5: Bootstrap starting.
I20260620 06:11:07.751763 745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:11:07.748114 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.194:38267 (local address 127.0.0.1:33729)
0620 06:11:07.748539 (+ 425us) server_negotiation.cc:206] Beginning negotiation
0620 06:11:07.748561 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0620 06:11:07.748595 (+ 34us) server_negotiation.cc:407] Connection header received
0620 06:11:07.749197 (+ 602us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:11:07.749213 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:11:07.749471 (+ 258us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:11:07.749826 (+ 355us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:11:07.750571 (+ 745us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:11:07.750575 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:11:07.750587 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 06:11:07.750597 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:11:07.750649 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 06:11:07.750667 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:11:07.750817 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0620 06:11:07.750886 (+ 69us) server_negotiation.cc:299] Negotiation successful
0620 06:11:07.751038 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":288,"thread_start_us":130,"threads_started":1}
I20260620 06:11:07.753849 742 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 443739ddf8f24361bf49f0b5fc726ed5: Neither blocks nor log segments found. Creating new log.
I20260620 06:11:07.754369 11672 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 06:11:07.754630 742 log.cc:824] T 00000000000000000000000000000000 P 443739ddf8f24361bf49f0b5fc726ed5: Log is configured to *not* fsync() on all Append() calls
I20260620 06:11:07.756490 742 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 443739ddf8f24361bf49f0b5fc726ed5: No bootstrap required, opened a new log
I20260620 06:11:07.757866 745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:11:07.756374 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.194:36483 (local address 127.0.0.1:33729)
0620 06:11:07.756494 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0620 06:11:07.756500 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 06:11:07.756698 (+ 198us) server_negotiation.cc:407] Connection header received
0620 06:11:07.756798 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:11:07.756801 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:11:07.756836 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:11:07.756911 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:11:07.757523 (+ 612us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:11:07.757527 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:11:07.757529 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 06:11:07.757540 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:11:07.757565 (+ 25us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 06:11:07.757577 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:11:07.757663 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0620 06:11:07.757718 (+ 55us) server_negotiation.cc:299] Negotiation successful
0620 06:11:07.757753 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260620 06:11:07.760459 742 raft_consensus.cc:348] T 00000000000000000000000000000000 P 443739ddf8f24361bf49f0b5fc726ed5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "443739ddf8f24361bf49f0b5fc726ed5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33729 } }
I20260620 06:11:07.760668 742 raft_consensus.cc:374] T 00000000000000000000000000000000 P 443739ddf8f24361bf49f0b5fc726ed5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 06:11:07.760710 742 raft_consensus.cc:729] T 00000000000000000000000000000000 P 443739ddf8f24361bf49f0b5fc726ed5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 443739ddf8f24361bf49f0b5fc726ed5, State: Initialized, Role: FOLLOWER
I20260620 06:11:07.761138 742 consensus_queue.cc:260] T 00000000000000000000000000000000 P 443739ddf8f24361bf49f0b5fc726ed5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "443739ddf8f24361bf49f0b5fc726ed5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33729 } }
I20260620 06:11:07.761273 742 raft_consensus.cc:388] T 00000000000000000000000000000000 P 443739ddf8f24361bf49f0b5fc726ed5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 06:11:07.761312 742 raft_consensus.cc:482] T 00000000000000000000000000000000 P 443739ddf8f24361bf49f0b5fc726ed5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 06:11:07.761382 742 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 443739ddf8f24361bf49f0b5fc726ed5 [term 0 FOLLOWER]: Advancing to term 1
I20260620 06:11:07.762164 742 raft_consensus.cc:504] T 00000000000000000000000000000000 P 443739ddf8f24361bf49f0b5fc726ed5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "443739ddf8f24361bf49f0b5fc726ed5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33729 } }
I20260620 06:11:07.762439 742 leader_election.cc:302] T 00000000000000000000000000000000 P 443739ddf8f24361bf49f0b5fc726ed5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 443739ddf8f24361bf49f0b5fc726ed5; no voters:
I20260620 06:11:07.762672 742 leader_election.cc:288] T 00000000000000000000000000000000 P 443739ddf8f24361bf49f0b5fc726ed5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 06:11:07.762811 752 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 443739ddf8f24361bf49f0b5fc726ed5 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 06:11:07.763136 752 raft_consensus.cc:686] T 00000000000000000000000000000000 P 443739ddf8f24361bf49f0b5fc726ed5 [term 1 LEADER]: Becoming Leader. State: Replica: 443739ddf8f24361bf49f0b5fc726ed5, State: Running, Role: LEADER
I20260620 06:11:07.763443 752 consensus_queue.cc:237] T 00000000000000000000000000000000 P 443739ddf8f24361bf49f0b5fc726ed5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "443739ddf8f24361bf49f0b5fc726ed5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33729 } }
I20260620 06:11:07.764006 742 sys_catalog.cc:564] T 00000000000000000000000000000000 P 443739ddf8f24361bf49f0b5fc726ed5 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 06:11:07.765064 754 sys_catalog.cc:455] T 00000000000000000000000000000000 P 443739ddf8f24361bf49f0b5fc726ed5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 443739ddf8f24361bf49f0b5fc726ed5. Latest consensus state: current_term: 1 leader_uuid: "443739ddf8f24361bf49f0b5fc726ed5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "443739ddf8f24361bf49f0b5fc726ed5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33729 } } }
I20260620 06:11:07.765202 754 sys_catalog.cc:458] T 00000000000000000000000000000000 P 443739ddf8f24361bf49f0b5fc726ed5 [sys.catalog]: This master's current role is: LEADER
I20260620 06:11:07.765614 761 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 06:11:07.765588 753 sys_catalog.cc:455] T 00000000000000000000000000000000 P 443739ddf8f24361bf49f0b5fc726ed5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "443739ddf8f24361bf49f0b5fc726ed5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "443739ddf8f24361bf49f0b5fc726ed5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33729 } } }
I20260620 06:11:07.765815 753 sys_catalog.cc:458] T 00000000000000000000000000000000 P 443739ddf8f24361bf49f0b5fc726ed5 [sys.catalog]: This master's current role is: LEADER
I20260620 06:11:07.768936 761 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 06:11:07.774005 761 catalog_manager.cc:1347] Generated new cluster ID: fc5314cd224b43979344fddcbcc40807
I20260620 06:11:07.774070 761 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 06:11:07.782341 761 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 06:11:07.782635 761 catalog_manager.cc:1504] Loading token signing keys...
I20260620 06:11:07.789995 761 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 443739ddf8f24361bf49f0b5fc726ed5: Generated new TSK 0
I20260620 06:11:07.790630 761 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 06:11:07.795503 11672 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9HMp43/build/debug/bin/kudu with pid 674
2026-06-20T06:11:07Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1373 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781935603120653-11672-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 06:11:07 dist-test-slave-9hm5 krb5kdc[773](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 06:11:07 dist-test-slave-9hm5 krb5kdc[773](info): set up 2 sockets
Jun 20 06:11:07 dist-test-slave-9hm5 krb5kdc[773](info): commencing operation
krb5kdc: starting...
W20260620 06:11:08.835928 11672 mini_kdc.cc:121] Time spent starting KDC: real 1.021s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781935603120653-11672-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781935603120653-11672-0/krb5kdc/test-user.keytab.
Jun 20 06:11:08 dist-test-slave-9hm5 krb5kdc[773](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935868, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T06:11:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T06:11:08Z Disabled control of system clock
WARNING: no policy specified for kudu/127.11.102.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.102.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.102.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781935603120653-11672-0/krb5kdc/kudu_127.11.102.62.keytab.
Entry for principal kudu/127.11.102.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781935603120653-11672-0/krb5kdc/kudu_127.11.102.62.keytab.
I20260620 06:11:08.954651 11672 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9HMp43/build/debug/bin/kudu
/tmp/dist-test-task9HMp43/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781935603120653-11672-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781935603120653-11672-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781935603120653-11672-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781935603120653-11672-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.102.62:36121
--webserver_interface=127.11.102.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781935603120653-11672-0/krb5kdc/kudu_127.11.102.62.keytab
--principal=kudu/127.11.102.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.102.20:34973
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.102.62:36121
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781935603120653-11672-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781935603120653-11672-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781935603120653-11672-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 06:11:09.064318 786 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:11:09.064592 786 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:11:09.064635 786 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:11:09.068681 786 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 06:11:09.068735 786 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:11:09.068755 786 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:11:09.068795 786 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 06:11:09.068825 786 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 06:11:09.073987 786 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.102.20:34973
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781935603120653-11672-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781935603120653-11672-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.102.62:36121
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781935603120653-11672-0/krb5kdc/kudu_127.11.102.62.keytab
--principal=kudu/127.11.102.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.102.62:36121
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781935603120653-11672-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.102.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.786
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781935603120653-11672-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 05:28:48 UTC on ccycloud.quasar-fixhms.root.comops.site
build id 4457020
I20260620 06:11:09.075057 786 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:11:09.076109 786 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 06:11:09.084520 794 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:11:09.084546 791 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:11:09.084522 792 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:11:09.084795 786 server_base.cc:1034] running on GCE node
I20260620 06:11:09.085482 786 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:11:09.086580 786 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:11:09.087751 786 hybrid_clock.cc:648] HybridClock initialized: now 1781935869087742 us; error 28 us; skew 500 ppm
Jun 20 06:11:09 dist-test-slave-9hm5 krb5kdc[773](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935869, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.102.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 06:11:09.090489 786 init.cc:377] Logged in from keytab as kudu/127.11.102.62@KRBTEST.COM (short username kudu)
I20260620 06:11:09.091368 786 webserver.cc:460] Webserver started at http://127.11.102.62:35351/ using document root <none> and password file <none>
I20260620 06:11:09.091985 786 fs_manager.cc:359] Metadata directory not provided
I20260620 06:11:09.092077 786 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:11:09.092309 786 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 06:11:09.094321 786 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781935603120653-11672-0/minicluster-data/master-0/data/instance:
uuid: "3597638c9d00483db4b8a5c2de336cb2"
format_stamp: "Formatted at 2026-06-20 06:11:09 on dist-test-slave-9hm5"
server_key: "237584776ad2a63c8ce580bc27f57032"
server_key_iv: "1b4af180923349f7e428cbe894d54ed2"
server_key_version: "encryptionkey@0"
I20260620 06:11:09.094871 786 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781935603120653-11672-0/minicluster-data/master-0/wal/instance:
uuid: "3597638c9d00483db4b8a5c2de336cb2"
format_stamp: "Formatted at 2026-06-20 06:11:09 on dist-test-slave-9hm5"
server_key: "237584776ad2a63c8ce580bc27f57032"
server_key_iv: "1b4af180923349f7e428cbe894d54ed2"
server_key_version: "encryptionkey@0"
I20260620 06:11:09.098608 786 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260620 06:11:09.101051 801 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:11:09.102303 786 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260620 06:11:09.102443 786 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781935603120653-11672-0/minicluster-data/master-0/data,/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781935603120653-11672-0/minicluster-data/master-0/wal
uuid: "3597638c9d00483db4b8a5c2de336cb2"
format_stamp: "Formatted at 2026-06-20 06:11:09 on dist-test-slave-9hm5"
server_key: "237584776ad2a63c8ce580bc27f57032"
server_key_iv: "1b4af180923349f7e428cbe894d54ed2"
server_key_version: "encryptionkey@0"
I20260620 06:11:09.102555 786 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781935603120653-11672-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781935603120653-11672-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781935603120653-11672-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:11:09.125589 786 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:11:09.128928 786 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:11:09.129142 786 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:11:09.136034 786 rpc_server.cc:307] RPC server started. Bound to: 127.11.102.62:36121
I20260620 06:11:09.136077 853 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.102.62:36121 every 8 connection(s)
I20260620 06:11:09.136989 786 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781935603120653-11672-0/minicluster-data/master-0/data/info.pb
I20260620 06:11:09.139665 854 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 06:11:09.142226 11672 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9HMp43/build/debug/bin/kudu as pid 786
I20260620 06:11:09.142387 11672 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781935603120653-11672-0/minicluster-data/master-0/wal/instance
I20260620 06:11:09.142661 11672 external_mini_cluster.cc:1428] Setting key 095fae5d40f88c16a6cfaa960ddf5a18
I20260620 06:11:09.145524 854 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3597638c9d00483db4b8a5c2de336cb2: Bootstrap starting.
I20260620 06:11:09.147743 854 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3597638c9d00483db4b8a5c2de336cb2: Neither blocks nor log segments found. Creating new log.
I20260620 06:11:09.148452 854 log.cc:824] T 00000000000000000000000000000000 P 3597638c9d00483db4b8a5c2de336cb2: Log is configured to *not* fsync() on all Append() calls
I20260620 06:11:09.150203 854 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3597638c9d00483db4b8a5c2de336cb2: No bootstrap required, opened a new log
Jun 20 06:11:09 dist-test-slave-9hm5 krb5kdc[773](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935868, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.102.62@KRBTEST.COM
I20260620 06:11:09.153770 854 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3597638c9d00483db4b8a5c2de336cb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3597638c9d00483db4b8a5c2de336cb2" member_type: VOTER last_known_addr { host: "127.11.102.62" port: 36121 } }
I20260620 06:11:09.153980 854 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3597638c9d00483db4b8a5c2de336cb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 06:11:09.154045 854 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3597638c9d00483db4b8a5c2de336cb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3597638c9d00483db4b8a5c2de336cb2, State: Initialized, Role: FOLLOWER
I20260620 06:11:09.154542 854 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3597638c9d00483db4b8a5c2de336cb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3597638c9d00483db4b8a5c2de336cb2" member_type: VOTER last_known_addr { host: "127.11.102.62" port: 36121 } }
I20260620 06:11:09.154713 854 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3597638c9d00483db4b8a5c2de336cb2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 06:11:09.154788 854 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3597638c9d00483db4b8a5c2de336cb2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 06:11:09.154917 854 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3597638c9d00483db4b8a5c2de336cb2 [term 0 FOLLOWER]: Advancing to term 1
I20260620 06:11:09.155977 854 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3597638c9d00483db4b8a5c2de336cb2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3597638c9d00483db4b8a5c2de336cb2" member_type: VOTER last_known_addr { host: "127.11.102.62" port: 36121 } }
I20260620 06:11:09.156347 854 leader_election.cc:302] T 00000000000000000000000000000000 P 3597638c9d00483db4b8a5c2de336cb2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3597638c9d00483db4b8a5c2de336cb2; no voters:
I20260620 06:11:09.156700 854 leader_election.cc:288] T 00000000000000000000000000000000 P 3597638c9d00483db4b8a5c2de336cb2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 06:11:09.156800 859 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3597638c9d00483db4b8a5c2de336cb2 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 06:11:09.156991 859 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3597638c9d00483db4b8a5c2de336cb2 [term 1 LEADER]: Becoming Leader. State: Replica: 3597638c9d00483db4b8a5c2de336cb2, State: Running, Role: LEADER
I20260620 06:11:09.157354 859 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3597638c9d00483db4b8a5c2de336cb2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3597638c9d00483db4b8a5c2de336cb2" member_type: VOTER last_known_addr { host: "127.11.102.62" port: 36121 } }
I20260620 06:11:09.157644 857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:11:09.144029 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59905 (local address 127.11.102.62:36121)
0620 06:11:09.144457 (+ 428us) server_negotiation.cc:206] Beginning negotiation
0620 06:11:09.144476 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0620 06:11:09.144509 (+ 33us) server_negotiation.cc:407] Connection header received
0620 06:11:09.145111 (+ 602us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:11:09.145127 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:11:09.145457 (+ 330us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:11:09.145796 (+ 339us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:11:09.146594 (+ 798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:11:09.147581 (+ 987us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:11:09.148515 (+ 934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:11:09.149017 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:11:09.149192 (+ 175us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:11:09.151947 (+ 2755us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:11:09.151952 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:11:09.151964 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:11:09.151989 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:11:09.154271 (+ 2282us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:11:09.154897 (+ 626us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:11:09.154902 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:11:09.154908 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:11:09.155070 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:11:09.155410 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:11:09.155412 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:11:09.155413 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:11:09.155853 (+ 440us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:11:09.156016 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0620 06:11:09.156334 (+ 318us) server_negotiation.cc:299] Negotiation successful
0620 06:11:09.156586 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":275,"thread_start_us":136,"threads_started":1}
I20260620 06:11:09.158038 854 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3597638c9d00483db4b8a5c2de336cb2 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 06:11:09.158496 859 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3597638c9d00483db4b8a5c2de336cb2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3597638c9d00483db4b8a5c2de336cb2. Latest consensus state: current_term: 1 leader_uuid: "3597638c9d00483db4b8a5c2de336cb2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3597638c9d00483db4b8a5c2de336cb2" member_type: VOTER last_known_addr { host: "127.11.102.62" port: 36121 } } }
I20260620 06:11:09.158663 859 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3597638c9d00483db4b8a5c2de336cb2 [sys.catalog]: This master's current role is: LEADER
I20260620 06:11:09.158932 860 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3597638c9d00483db4b8a5c2de336cb2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3597638c9d00483db4b8a5c2de336cb2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3597638c9d00483db4b8a5c2de336cb2" member_type: VOTER last_known_addr { host: "127.11.102.62" port: 36121 } } }
I20260620 06:11:09.159051 864 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 06:11:09.159101 860 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3597638c9d00483db4b8a5c2de336cb2 [sys.catalog]: This master's current role is: LEADER
I20260620 06:11:09.160833 11672 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 06:11:09.162460 864 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 06:11:09.167876 864 catalog_manager.cc:1347] Generated new cluster ID: 1db0a04c22c345e499b3af35957c8c10
I20260620 06:11:09.167932 864 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 06:11:09.174232 857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:11:09.164052 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35653 (local address 127.11.102.62:36121)
0620 06:11:09.164149 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0620 06:11:09.164152 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 06:11:09.164487 (+ 335us) server_negotiation.cc:407] Connection header received
0620 06:11:09.164622 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:11:09.164625 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:11:09.164672 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:11:09.164761 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:11:09.165564 (+ 803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:11:09.166360 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:11:09.167625 (+ 1265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:11:09.168052 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:11:09.168151 (+ 99us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:11:09.169994 (+ 1843us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:11:09.169998 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:11:09.170001 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:11:09.170027 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:11:09.171806 (+ 1779us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:11:09.172403 (+ 597us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:11:09.172406 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:11:09.172409 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:11:09.172490 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:11:09.172884 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:11:09.172888 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:11:09.172890 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:11:09.173096 (+ 206us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:11:09.173252 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0620 06:11:09.173971 (+ 719us) server_negotiation.cc:299] Negotiation successful
0620 06:11:09.174068 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260620 06:11:09.178396 864 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 06:11:09.179307 864 catalog_manager.cc:1504] Loading token signing keys...
I20260620 06:11:09.183530 864 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3597638c9d00483db4b8a5c2de336cb2: Generated new TSK 0
I20260620 06:11:09.184357 864 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 06:11:09.194408 11672 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9HMp43/build/debug/bin/kudu with pid 786
W20260620 06:11:09.199565 771 connection.cc:535] client connection to 127.11.102.62:36121 recv error: Network error: failed to read from TLS socket (remote: 127.11.102.62:36121): Connection reset by peer (error 104)
2026-06-20T06:11:09Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (1399 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781935603120653-11672-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 06:11:09 dist-test-slave-9hm5 krb5kdc[885](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 06:11:09 dist-test-slave-9hm5 krb5kdc[885](info): set up 2 sockets
Jun 20 06:11:09 dist-test-slave-9hm5 krb5kdc[885](info): commencing operation
krb5kdc: starting...
W20260620 06:11:10.227898 11672 mini_kdc.cc:121] Time spent starting KDC: real 1.014s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781935603120653-11672-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781935603120653-11672-0/krb5kdc/test-user.keytab.
Jun 20 06:11:10 dist-test-slave-9hm5 krb5kdc[885](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935870, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T06:11:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T06:11:10Z Disabled control of system clock
WARNING: no policy specified for kudu/127.11.102.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.102.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.102.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781935603120653-11672-0/krb5kdc/kudu_127.11.102.62.keytab.
Entry for principal kudu/127.11.102.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781935603120653-11672-0/krb5kdc/kudu_127.11.102.62.keytab.
I20260620 06:11:10.350524 11672 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9HMp43/build/debug/bin/kudu
/tmp/dist-test-task9HMp43/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781935603120653-11672-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781935603120653-11672-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781935603120653-11672-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781935603120653-11672-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.102.62:39769
--webserver_interface=127.11.102.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781935603120653-11672-0/krb5kdc/kudu_127.11.102.62.keytab
--principal=kudu/127.11.102.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.102.20:36837
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.102.62:39769
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781935603120653-11672-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781935603120653-11672-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781935603120653-11672-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 06:11:10.460337 898 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:11:10.460628 898 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:11:10.460671 898 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:11:10.464717 898 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 06:11:10.464788 898 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:11:10.464809 898 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:11:10.464826 898 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 06:11:10.464841 898 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 06:11:10.470330 898 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.102.20:36837
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781935603120653-11672-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781935603120653-11672-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.102.62:39769
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781935603120653-11672-0/krb5kdc/kudu_127.11.102.62.keytab
--principal=kudu/127.11.102.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.102.62:39769
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781935603120653-11672-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.102.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.898
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781935603120653-11672-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 05:28:48 UTC on ccycloud.quasar-fixhms.root.comops.site
build id 4457020
I20260620 06:11:10.471431 898 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:11:10.472496 898 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 06:11:10.480787 906 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:11:10.480822 898 server_base.cc:1034] running on GCE node
W20260620 06:11:10.480787 904 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:11:10.480787 903 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:11:10.481634 898 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:11:10.482707 898 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:11:10.483887 898 hybrid_clock.cc:648] HybridClock initialized: now 1781935870483858 us; error 41 us; skew 500 ppm
Jun 20 06:11:10 dist-test-slave-9hm5 krb5kdc[885](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935870, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.102.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 06:11:10.486747 898 init.cc:377] Logged in from keytab as kudu/127.11.102.62@KRBTEST.COM (short username kudu)
I20260620 06:11:10.487748 898 webserver.cc:460] Webserver started at http://127.11.102.62:42235/ using document root <none> and password file <none>
I20260620 06:11:10.488325 898 fs_manager.cc:359] Metadata directory not provided
I20260620 06:11:10.488417 898 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:11:10.488651 898 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 06:11:10.490667 898 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781935603120653-11672-0/minicluster-data/master-0/data/instance:
uuid: "2aa66d57cee94cacad4690c3f0a2fba0"
format_stamp: "Formatted at 2026-06-20 06:11:10 on dist-test-slave-9hm5"
server_key: "7bde75a7d8641de67791adcb2d19fd74"
server_key_iv: "dbd1a2f93896964779bc00b2e9df5137"
server_key_version: "encryptionkey@0"
I20260620 06:11:10.491233 898 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781935603120653-11672-0/minicluster-data/master-0/wal/instance:
uuid: "2aa66d57cee94cacad4690c3f0a2fba0"
format_stamp: "Formatted at 2026-06-20 06:11:10 on dist-test-slave-9hm5"
server_key: "7bde75a7d8641de67791adcb2d19fd74"
server_key_iv: "dbd1a2f93896964779bc00b2e9df5137"
server_key_version: "encryptionkey@0"
I20260620 06:11:10.494820 898 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.003s
I20260620 06:11:10.497342 913 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:11:10.498546 898 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260620 06:11:10.498692 898 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781935603120653-11672-0/minicluster-data/master-0/data,/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781935603120653-11672-0/minicluster-data/master-0/wal
uuid: "2aa66d57cee94cacad4690c3f0a2fba0"
format_stamp: "Formatted at 2026-06-20 06:11:10 on dist-test-slave-9hm5"
server_key: "7bde75a7d8641de67791adcb2d19fd74"
server_key_iv: "dbd1a2f93896964779bc00b2e9df5137"
server_key_version: "encryptionkey@0"
I20260620 06:11:10.498808 898 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781935603120653-11672-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781935603120653-11672-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781935603120653-11672-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:11:10.509239 898 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:11:10.512652 898 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:11:10.512864 898 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:11:10.520166 898 rpc_server.cc:307] RPC server started. Bound to: 127.11.102.62:39769
I20260620 06:11:10.520177 965 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.102.62:39769 every 8 connection(s)
I20260620 06:11:10.521126 898 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781935603120653-11672-0/minicluster-data/master-0/data/info.pb
I20260620 06:11:10.523831 966 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 06:11:10.526808 11672 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9HMp43/build/debug/bin/kudu as pid 898
I20260620 06:11:10.526923 11672 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781935603120653-11672-0/minicluster-data/master-0/wal/instance
I20260620 06:11:10.527196 11672 external_mini_cluster.cc:1428] Setting key 51f45f8df24e37cc5dbb87e10733d75e
I20260620 06:11:10.529690 966 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2aa66d57cee94cacad4690c3f0a2fba0: Bootstrap starting.
I20260620 06:11:10.531912 966 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2aa66d57cee94cacad4690c3f0a2fba0: Neither blocks nor log segments found. Creating new log.
I20260620 06:11:10.532655 966 log.cc:824] T 00000000000000000000000000000000 P 2aa66d57cee94cacad4690c3f0a2fba0: Log is configured to *not* fsync() on all Append() calls
I20260620 06:11:10.534454 966 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2aa66d57cee94cacad4690c3f0a2fba0: No bootstrap required, opened a new log
I20260620 06:11:10.536038 969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:11:10.528646 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46121 (local address 127.11.102.62:39769)
0620 06:11:10.529080 (+ 434us) server_negotiation.cc:206] Beginning negotiation
0620 06:11:10.529095 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 06:11:10.529128 (+ 33us) server_negotiation.cc:407] Connection header received
0620 06:11:10.529720 (+ 592us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:11:10.529735 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:11:10.530120 (+ 385us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:11:10.530456 (+ 336us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:11:10.530857 (+ 401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:11:10.531905 (+ 1048us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:11:10.532790 (+ 885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:11:10.533262 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:11:10.533425 (+ 163us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:11:10.534312 (+ 887us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:11:10.534317 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:11:10.534329 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 06:11:10.534751 (+ 422us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:11:10.534802 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 06:11:10.534825 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:11:10.534959 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0620 06:11:10.535136 (+ 177us) server_negotiation.cc:299] Negotiation successful
0620 06:11:10.535279 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":296,"thread_start_us":136,"threads_started":1}
I20260620 06:11:10.538481 966 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2aa66d57cee94cacad4690c3f0a2fba0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2aa66d57cee94cacad4690c3f0a2fba0" member_type: VOTER last_known_addr { host: "127.11.102.62" port: 39769 } }
I20260620 06:11:10.538748 966 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2aa66d57cee94cacad4690c3f0a2fba0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 06:11:10.538851 966 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2aa66d57cee94cacad4690c3f0a2fba0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2aa66d57cee94cacad4690c3f0a2fba0, State: Initialized, Role: FOLLOWER
I20260620 06:11:10.538982 11672 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 06:11:10.539323 966 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2aa66d57cee94cacad4690c3f0a2fba0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2aa66d57cee94cacad4690c3f0a2fba0" member_type: VOTER last_known_addr { host: "127.11.102.62" port: 39769 } }
I20260620 06:11:10.539474 966 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2aa66d57cee94cacad4690c3f0a2fba0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 06:11:10.539559 966 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2aa66d57cee94cacad4690c3f0a2fba0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 06:11:10.539687 966 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2aa66d57cee94cacad4690c3f0a2fba0 [term 0 FOLLOWER]: Advancing to term 1
I20260620 06:11:10.540706 966 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2aa66d57cee94cacad4690c3f0a2fba0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2aa66d57cee94cacad4690c3f0a2fba0" member_type: VOTER last_known_addr { host: "127.11.102.62" port: 39769 } }
I20260620 06:11:10.541093 966 leader_election.cc:302] T 00000000000000000000000000000000 P 2aa66d57cee94cacad4690c3f0a2fba0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2aa66d57cee94cacad4690c3f0a2fba0; no voters:
I20260620 06:11:10.541442 966 leader_election.cc:288] T 00000000000000000000000000000000 P 2aa66d57cee94cacad4690c3f0a2fba0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 06:11:10.541527 976 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2aa66d57cee94cacad4690c3f0a2fba0 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 06:11:10.541805 976 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2aa66d57cee94cacad4690c3f0a2fba0 [term 1 LEADER]: Becoming Leader. State: Replica: 2aa66d57cee94cacad4690c3f0a2fba0, State: Running, Role: LEADER
I20260620 06:11:10.542191 976 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2aa66d57cee94cacad4690c3f0a2fba0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2aa66d57cee94cacad4690c3f0a2fba0" member_type: VOTER last_known_addr { host: "127.11.102.62" port: 39769 } }
I20260620 06:11:10.542760 966 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2aa66d57cee94cacad4690c3f0a2fba0 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 06:11:10.543910 977 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2aa66d57cee94cacad4690c3f0a2fba0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2aa66d57cee94cacad4690c3f0a2fba0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2aa66d57cee94cacad4690c3f0a2fba0" member_type: VOTER last_known_addr { host: "127.11.102.62" port: 39769 } } }
I20260620 06:11:10.544088 977 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2aa66d57cee94cacad4690c3f0a2fba0 [sys.catalog]: This master's current role is: LEADER
I20260620 06:11:10.544443 977 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2aa66d57cee94cacad4690c3f0a2fba0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2aa66d57cee94cacad4690c3f0a2fba0. Latest consensus state: current_term: 1 leader_uuid: "2aa66d57cee94cacad4690c3f0a2fba0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2aa66d57cee94cacad4690c3f0a2fba0" member_type: VOTER last_known_addr { host: "127.11.102.62" port: 39769 } } }
I20260620 06:11:10.544574 977 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2aa66d57cee94cacad4690c3f0a2fba0 [sys.catalog]: This master's current role is: LEADER
I20260620 06:11:10.546131 969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:11:10.541099 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44915 (local address 127.11.102.62:39769)
0620 06:11:10.541222 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0620 06:11:10.541228 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 06:11:10.541500 (+ 272us) server_negotiation.cc:407] Connection header received
0620 06:11:10.541638 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:11:10.541644 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:11:10.541695 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:11:10.541803 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:11:10.542090 (+ 287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:11:10.542853 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:11:10.544080 (+ 1227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:11:10.544487 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:11:10.544589 (+ 102us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:11:10.545720 (+ 1131us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:11:10.545723 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:11:10.545725 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 06:11:10.545738 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:11:10.545764 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 06:11:10.545776 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:11:10.545861 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0620 06:11:10.545973 (+ 112us) server_negotiation.cc:299] Negotiation successful
0620 06:11:10.546010 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
W20260620 06:11:10.548273 991 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 2aa66d57cee94cacad4690c3f0a2fba0: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 06:11:10.548347 991 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 06:11:10.548401 987 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 06:11:10.549228 987 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 06:11:10.554808 987 catalog_manager.cc:1347] Generated new cluster ID: 87d6382aa549467b9531d08206ef7c8a
I20260620 06:11:10.554879 987 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 06:11:10.562552 987 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 06:11:10.563791 987 catalog_manager.cc:1504] Loading token signing keys...
I20260620 06:11:10.576582 987 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2aa66d57cee94cacad4690c3f0a2fba0: Generated new TSK 0
I20260620 06:11:10.577489 987 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 06:11:10.585073 11672 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9HMp43/build/debug/bin/kudu with pid 898
W20260620 06:11:10.590421 883 connection.cc:535] client connection to 127.11.102.62:39769 recv error: Network error: failed to read from TLS socket (remote: 127.11.102.62:39769): Connection reset by peer (error 104)
2026-06-20T06:11:10Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1390 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781935603120653-11672-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 06:11:10 dist-test-slave-9hm5 krb5kdc[997](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 06:11:10 dist-test-slave-9hm5 krb5kdc[997](info): set up 2 sockets
Jun 20 06:11:10 dist-test-slave-9hm5 krb5kdc[997](info): commencing operation
krb5kdc: starting...
W20260620 06:11:11.631968 11672 mini_kdc.cc:121] Time spent starting KDC: real 1.027s user 0.006s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781935603120653-11672-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781935603120653-11672-0/krb5kdc/test-user.keytab.
Jun 20 06:11:11 dist-test-slave-9hm5 krb5kdc[997](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935871, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T06:11:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T06:11:11Z Disabled control of system clock
WARNING: no policy specified for kudu/127.11.102.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.102.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.102.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781935603120653-11672-0/krb5kdc/kudu_127.11.102.62.keytab.
Entry for principal kudu/127.11.102.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781935603120653-11672-0/krb5kdc/kudu_127.11.102.62.keytab.
I20260620 06:11:11.758682 11672 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9HMp43/build/debug/bin/kudu
/tmp/dist-test-task9HMp43/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781935603120653-11672-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781935603120653-11672-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781935603120653-11672-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781935603120653-11672-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.102.62:36845
--webserver_interface=127.11.102.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781935603120653-11672-0/krb5kdc/kudu_127.11.102.62.keytab
--principal=kudu/127.11.102.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.102.20:44629
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.102.62:36845
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781935603120653-11672-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781935603120653-11672-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781935603120653-11672-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 06:11:11.869729 1010 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:11:11.870020 1010 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:11:11.870062 1010 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:11:11.874137 1010 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 06:11:11.874207 1010 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:11:11.874226 1010 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:11:11.874244 1010 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 06:11:11.874259 1010 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 06:11:11.879664 1010 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.102.20:44629
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781935603120653-11672-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781935603120653-11672-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.102.62:36845
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781935603120653-11672-0/krb5kdc/kudu_127.11.102.62.keytab
--principal=kudu/127.11.102.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.102.62:36845
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781935603120653-11672-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.102.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1010
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781935603120653-11672-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 05:28:48 UTC on ccycloud.quasar-fixhms.root.comops.site
build id 4457020
I20260620 06:11:11.880785 1010 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:11:11.881788 1010 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 06:11:11.890771 1015 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:11:11.890771 1018 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:11:11.890769 1016 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:11:11.890851 1010 server_base.cc:1034] running on GCE node
I20260620 06:11:11.891686 1010 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:11:11.892802 1010 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:11:11.894016 1010 hybrid_clock.cc:648] HybridClock initialized: now 1781935871893967 us; error 60 us; skew 500 ppm
Jun 20 06:11:11 dist-test-slave-9hm5 krb5kdc[997](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935871, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.102.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 06:11:11.896781 1010 init.cc:377] Logged in from keytab as kudu/127.11.102.62@KRBTEST.COM (short username kudu)
I20260620 06:11:11.897701 1010 webserver.cc:460] Webserver started at http://127.11.102.62:35971/ using document root <none> and password file <none>
I20260620 06:11:11.898221 1010 fs_manager.cc:359] Metadata directory not provided
I20260620 06:11:11.898279 1010 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:11:11.898449 1010 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 06:11:11.900528 1010 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781935603120653-11672-0/minicluster-data/master-0/data/instance:
uuid: "872eb9cd4a8a4d1f8fd8348679560a19"
format_stamp: "Formatted at 2026-06-20 06:11:11 on dist-test-slave-9hm5"
server_key: "73541d9fc82c2bdad683334823fc0bc4"
server_key_iv: "5495dbbf8f72dcfed103f185c4a8dd4c"
server_key_version: "encryptionkey@0"
I20260620 06:11:11.901017 1010 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781935603120653-11672-0/minicluster-data/master-0/wal/instance:
uuid: "872eb9cd4a8a4d1f8fd8348679560a19"
format_stamp: "Formatted at 2026-06-20 06:11:11 on dist-test-slave-9hm5"
server_key: "73541d9fc82c2bdad683334823fc0bc4"
server_key_iv: "5495dbbf8f72dcfed103f185c4a8dd4c"
server_key_version: "encryptionkey@0"
I20260620 06:11:11.904731 1010 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 06:11:11.907168 1025 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:11:11.908551 1010 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.003s
I20260620 06:11:11.908658 1010 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781935603120653-11672-0/minicluster-data/master-0/data,/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781935603120653-11672-0/minicluster-data/master-0/wal
uuid: "872eb9cd4a8a4d1f8fd8348679560a19"
format_stamp: "Formatted at 2026-06-20 06:11:11 on dist-test-slave-9hm5"
server_key: "73541d9fc82c2bdad683334823fc0bc4"
server_key_iv: "5495dbbf8f72dcfed103f185c4a8dd4c"
server_key_version: "encryptionkey@0"
I20260620 06:11:11.908752 1010 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781935603120653-11672-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781935603120653-11672-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781935603120653-11672-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:11:11.913344 1010 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:11:11.916492 1010 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:11:11.916651 1010 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:11:11.924031 1010 rpc_server.cc:307] RPC server started. Bound to: 127.11.102.62:36845
I20260620 06:11:11.924054 1077 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.102.62:36845 every 8 connection(s)
I20260620 06:11:11.925002 1010 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781935603120653-11672-0/minicluster-data/master-0/data/info.pb
I20260620 06:11:11.927680 1078 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 06:11:11.933171 1078 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 872eb9cd4a8a4d1f8fd8348679560a19: Bootstrap starting.
I20260620 06:11:11.934751 11672 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9HMp43/build/debug/bin/kudu as pid 1010
I20260620 06:11:11.934885 11672 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781935603120653-11672-0/minicluster-data/master-0/wal/instance
I20260620 06:11:11.935171 11672 external_mini_cluster.cc:1428] Setting key 597e37b5e20601f0fca9196209d621ee
I20260620 06:11:11.936254 1078 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 872eb9cd4a8a4d1f8fd8348679560a19: Neither blocks nor log segments found. Creating new log.
I20260620 06:11:11.937090 1078 log.cc:824] T 00000000000000000000000000000000 P 872eb9cd4a8a4d1f8fd8348679560a19: Log is configured to *not* fsync() on all Append() calls
I20260620 06:11:11.939150 1078 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 872eb9cd4a8a4d1f8fd8348679560a19: No bootstrap required, opened a new log
I20260620 06:11:11.940613 1081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:11:11.936383 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38909 (local address 127.11.102.62:36845)
0620 06:11:11.936864 (+ 481us) server_negotiation.cc:206] Beginning negotiation
0620 06:11:11.936878 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 06:11:11.936914 (+ 36us) server_negotiation.cc:407] Connection header received
0620 06:11:11.937766 (+ 852us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:11:11.937792 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:11:11.938108 (+ 316us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:11:11.938447 (+ 339us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:11:11.939246 (+ 799us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:11:11.939253 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:11:11.939270 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 06:11:11.939283 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:11:11.939346 (+ 63us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 06:11:11.939367 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:11:11.939481 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0620 06:11:11.939592 (+ 111us) server_negotiation.cc:299] Negotiation successful
0620 06:11:11.939911 (+ 319us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":330,"thread_start_us":151,"threads_started":1}
I20260620 06:11:11.942802 1078 raft_consensus.cc:348] T 00000000000000000000000000000000 P 872eb9cd4a8a4d1f8fd8348679560a19 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "872eb9cd4a8a4d1f8fd8348679560a19" member_type: VOTER last_known_addr { host: "127.11.102.62" port: 36845 } }
I20260620 06:11:11.943048 1078 raft_consensus.cc:374] T 00000000000000000000000000000000 P 872eb9cd4a8a4d1f8fd8348679560a19 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 06:11:11.943133 1078 raft_consensus.cc:729] T 00000000000000000000000000000000 P 872eb9cd4a8a4d1f8fd8348679560a19 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 872eb9cd4a8a4d1f8fd8348679560a19, State: Initialized, Role: FOLLOWER
I20260620 06:11:11.943333 11672 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 06:11:11.943600 1078 consensus_queue.cc:260] T 00000000000000000000000000000000 P 872eb9cd4a8a4d1f8fd8348679560a19 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "872eb9cd4a8a4d1f8fd8348679560a19" member_type: VOTER last_known_addr { host: "127.11.102.62" port: 36845 } }
I20260620 06:11:11.943773 1078 raft_consensus.cc:388] T 00000000000000000000000000000000 P 872eb9cd4a8a4d1f8fd8348679560a19 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 06:11:11.943846 1078 raft_consensus.cc:482] T 00000000000000000000000000000000 P 872eb9cd4a8a4d1f8fd8348679560a19 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 06:11:11.943953 1078 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 872eb9cd4a8a4d1f8fd8348679560a19 [term 0 FOLLOWER]: Advancing to term 1
I20260620 06:11:11.945242 1078 raft_consensus.cc:504] T 00000000000000000000000000000000 P 872eb9cd4a8a4d1f8fd8348679560a19 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "872eb9cd4a8a4d1f8fd8348679560a19" member_type: VOTER last_known_addr { host: "127.11.102.62" port: 36845 } }
I20260620 06:11:11.945618 1078 leader_election.cc:302] T 00000000000000000000000000000000 P 872eb9cd4a8a4d1f8fd8348679560a19 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 872eb9cd4a8a4d1f8fd8348679560a19; no voters:
I20260620 06:11:11.945961 1078 leader_election.cc:288] T 00000000000000000000000000000000 P 872eb9cd4a8a4d1f8fd8348679560a19 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 06:11:11.946079 1088 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 872eb9cd4a8a4d1f8fd8348679560a19 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 06:11:11.946329 1088 raft_consensus.cc:686] T 00000000000000000000000000000000 P 872eb9cd4a8a4d1f8fd8348679560a19 [term 1 LEADER]: Becoming Leader. State: Replica: 872eb9cd4a8a4d1f8fd8348679560a19, State: Running, Role: LEADER
I20260620 06:11:11.946667 1088 consensus_queue.cc:237] T 00000000000000000000000000000000 P 872eb9cd4a8a4d1f8fd8348679560a19 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "872eb9cd4a8a4d1f8fd8348679560a19" member_type: VOTER last_known_addr { host: "127.11.102.62" port: 36845 } }
I20260620 06:11:11.947365 1078 sys_catalog.cc:564] T 00000000000000000000000000000000 P 872eb9cd4a8a4d1f8fd8348679560a19 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 06:11:11.947395 1081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:11:11.945698 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60297 (local address 127.11.102.62:36845)
0620 06:11:11.945813 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0620 06:11:11.945818 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 06:11:11.946021 (+ 203us) server_negotiation.cc:407] Connection header received
0620 06:11:11.946148 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:11:11.946151 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:11:11.946192 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:11:11.946285 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:11:11.947011 (+ 726us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:11:11.947015 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:11:11.947018 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 06:11:11.947030 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:11:11.947057 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 06:11:11.947070 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:11:11.947156 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0620 06:11:11.947247 (+ 91us) server_negotiation.cc:299] Negotiation successful
0620 06:11:11.947283 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260620 06:11:11.947995 1090 sys_catalog.cc:455] T 00000000000000000000000000000000 P 872eb9cd4a8a4d1f8fd8348679560a19 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 872eb9cd4a8a4d1f8fd8348679560a19. Latest consensus state: current_term: 1 leader_uuid: "872eb9cd4a8a4d1f8fd8348679560a19" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "872eb9cd4a8a4d1f8fd8348679560a19" member_type: VOTER last_known_addr { host: "127.11.102.62" port: 36845 } } }
I20260620 06:11:11.948169 1090 sys_catalog.cc:458] T 00000000000000000000000000000000 P 872eb9cd4a8a4d1f8fd8348679560a19 [sys.catalog]: This master's current role is: LEADER
I20260620 06:11:11.948539 1089 sys_catalog.cc:455] T 00000000000000000000000000000000 P 872eb9cd4a8a4d1f8fd8348679560a19 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "872eb9cd4a8a4d1f8fd8348679560a19" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "872eb9cd4a8a4d1f8fd8348679560a19" member_type: VOTER last_known_addr { host: "127.11.102.62" port: 36845 } } }
I20260620 06:11:11.948647 1089 sys_catalog.cc:458] T 00000000000000000000000000000000 P 872eb9cd4a8a4d1f8fd8348679560a19 [sys.catalog]: This master's current role is: LEADER
W20260620 06:11:11.952426 1103 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 872eb9cd4a8a4d1f8fd8348679560a19: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 06:11:11.952512 1103 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 06:11:11.952569 1095 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 06:11:11.953400 1095 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 06:11:11.959208 1095 catalog_manager.cc:1347] Generated new cluster ID: ec9125c3296b493ca93132062be0d4ea
I20260620 06:11:11.959290 1095 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 06:11:11.990103 1095 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 06:11:11.990464 1095 catalog_manager.cc:1504] Loading token signing keys...
I20260620 06:11:11.998838 1095 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 872eb9cd4a8a4d1f8fd8348679560a19: Generated new TSK 0
I20260620 06:11:11.999550 1095 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 06:11:12.011022 11672 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9HMp43/build/debug/bin/kudu with pid 1010
2026-06-20T06:11:12Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1426 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781935603120653-11672-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 06:11:12 dist-test-slave-9hm5 krb5kdc[1109](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 06:11:12 dist-test-slave-9hm5 krb5kdc[1109](info): set up 2 sockets
Jun 20 06:11:12 dist-test-slave-9hm5 krb5kdc[1109](info): commencing operation
krb5kdc: starting...
W20260620 06:11:13.055974 11672 mini_kdc.cc:121] Time spent starting KDC: real 1.025s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781935603120653-11672-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781935603120653-11672-0/krb5kdc/test-user.keytab.
Jun 20 06:11:13 dist-test-slave-9hm5 krb5kdc[1109](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935873, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T06:11:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T06:11:13Z Disabled control of system clock
WARNING: no policy specified for kudu/127.11.102.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.102.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.102.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781935603120653-11672-0/krb5kdc/kudu_127.11.102.62.keytab.
Entry for principal kudu/127.11.102.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781935603120653-11672-0/krb5kdc/kudu_127.11.102.62.keytab.
I20260620 06:11:13.179064 11672 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9HMp43/build/debug/bin/kudu
/tmp/dist-test-task9HMp43/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781935603120653-11672-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781935603120653-11672-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781935603120653-11672-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781935603120653-11672-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.102.62:45603
--webserver_interface=127.11.102.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781935603120653-11672-0/krb5kdc/kudu_127.11.102.62.keytab
--principal=kudu/127.11.102.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.102.20:34483
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.102.62:45603
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781935603120653-11672-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781935603120653-11672-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781935603120653-11672-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 06:11:13.290251 1122 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:11:13.290560 1122 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:11:13.290635 1122 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:11:13.294656 1122 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 06:11:13.294761 1122 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:11:13.294798 1122 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:11:13.294837 1122 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 06:11:13.294898 1122 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 06:11:13.300262 1122 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.102.20:34483
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781935603120653-11672-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781935603120653-11672-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.102.62:45603
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781935603120653-11672-0/krb5kdc/kudu_127.11.102.62.keytab
--principal=kudu/127.11.102.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.102.62:45603
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781935603120653-11672-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.102.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1122
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781935603120653-11672-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 05:28:48 UTC on ccycloud.quasar-fixhms.root.comops.site
build id 4457020
I20260620 06:11:13.301685 1122 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:11:13.303057 1122 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 06:11:13.313035 1130 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:11:13.313161 1122 server_base.cc:1034] running on GCE node
W20260620 06:11:13.313043 1127 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:11:13.313043 1128 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:11:13.313889 1122 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:11:13.314960 1122 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:11:13.316143 1122 hybrid_clock.cc:648] HybridClock initialized: now 1781935873316134 us; error 30 us; skew 500 ppm
Jun 20 06:11:13 dist-test-slave-9hm5 krb5kdc[1109](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935873, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.102.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 06:11:13.322724 1122 init.cc:377] Logged in from keytab as kudu/127.11.102.62@KRBTEST.COM (short username kudu)
I20260620 06:11:13.323755 1122 webserver.cc:460] Webserver started at http://127.11.102.62:40875/ using document root <none> and password file <none>
I20260620 06:11:13.324333 1122 fs_manager.cc:359] Metadata directory not provided
I20260620 06:11:13.324421 1122 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:11:13.324653 1122 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 06:11:13.326663 1122 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781935603120653-11672-0/minicluster-data/master-0/data/instance:
uuid: "70868dce29ee4f89ace46943797eda44"
format_stamp: "Formatted at 2026-06-20 06:11:13 on dist-test-slave-9hm5"
server_key: "e120243188b8737e16667791faa05118"
server_key_iv: "5619de6a9c1d79617fdec36da361311c"
server_key_version: "encryptionkey@0"
I20260620 06:11:13.327237 1122 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781935603120653-11672-0/minicluster-data/master-0/wal/instance:
uuid: "70868dce29ee4f89ace46943797eda44"
format_stamp: "Formatted at 2026-06-20 06:11:13 on dist-test-slave-9hm5"
server_key: "e120243188b8737e16667791faa05118"
server_key_iv: "5619de6a9c1d79617fdec36da361311c"
server_key_version: "encryptionkey@0"
I20260620 06:11:13.331333 1122 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260620 06:11:13.334599 1137 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:11:13.336261 1122 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 06:11:13.336393 1122 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781935603120653-11672-0/minicluster-data/master-0/data,/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781935603120653-11672-0/minicluster-data/master-0/wal
uuid: "70868dce29ee4f89ace46943797eda44"
format_stamp: "Formatted at 2026-06-20 06:11:13 on dist-test-slave-9hm5"
server_key: "e120243188b8737e16667791faa05118"
server_key_iv: "5619de6a9c1d79617fdec36da361311c"
server_key_version: "encryptionkey@0"
I20260620 06:11:13.336511 1122 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781935603120653-11672-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781935603120653-11672-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781935603120653-11672-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:11:13.351096 1122 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:11:13.354234 1122 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:11:13.354462 1122 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:11:13.361809 1122 rpc_server.cc:307] RPC server started. Bound to: 127.11.102.62:45603
I20260620 06:11:13.361824 1189 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.102.62:45603 every 8 connection(s)
I20260620 06:11:13.362810 1122 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781935603120653-11672-0/minicluster-data/master-0/data/info.pb
I20260620 06:11:13.365407 11672 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9HMp43/build/debug/bin/kudu as pid 1122
I20260620 06:11:13.365540 11672 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781935603120653-11672-0/minicluster-data/master-0/wal/instance
I20260620 06:11:13.365760 1190 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 06:11:13.365878 11672 external_mini_cluster.cc:1428] Setting key cb0a0e1ba29259543c4c5dbbd08a7b32
I20260620 06:11:13.372129 1190 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 70868dce29ee4f89ace46943797eda44: Bootstrap starting.
I20260620 06:11:13.374516 1190 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 70868dce29ee4f89ace46943797eda44: Neither blocks nor log segments found. Creating new log.
Jun 20 06:11:13 dist-test-slave-9hm5 krb5kdc[1109](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935873, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.102.62@KRBTEST.COM
I20260620 06:11:13.375422 1190 log.cc:824] T 00000000000000000000000000000000 P 70868dce29ee4f89ace46943797eda44: Log is configured to *not* fsync() on all Append() calls
I20260620 06:11:13.377535 1190 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 70868dce29ee4f89ace46943797eda44: No bootstrap required, opened a new log
I20260620 06:11:13.380059 1193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:11:13.367277 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33813 (local address 127.11.102.62:45603)
0620 06:11:13.367806 (+ 529us) server_negotiation.cc:206] Beginning negotiation
0620 06:11:13.367821 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 06:11:13.367855 (+ 34us) server_negotiation.cc:407] Connection header received
0620 06:11:13.368457 (+ 602us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:11:13.368473 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:11:13.368803 (+ 330us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:11:13.369045 (+ 242us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:11:13.370173 (+ 1128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:11:13.371181 (+ 1008us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:11:13.372125 (+ 944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:11:13.372644 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:11:13.372821 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:11:13.375564 (+ 2743us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:11:13.375570 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:11:13.375583 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:11:13.375646 (+ 63us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:11:13.377290 (+ 1644us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:11:13.377770 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:11:13.377775 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:11:13.377779 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:11:13.377884 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:11:13.378166 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:11:13.378168 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:11:13.378169 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:11:13.378538 (+ 369us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:11:13.378674 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0620 06:11:13.378964 (+ 290us) server_negotiation.cc:299] Negotiation successful
0620 06:11:13.379255 (+ 291us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":385,"thread_start_us":158,"threads_started":1}
I20260620 06:11:13.381470 1190 raft_consensus.cc:348] T 00000000000000000000000000000000 P 70868dce29ee4f89ace46943797eda44 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70868dce29ee4f89ace46943797eda44" member_type: VOTER last_known_addr { host: "127.11.102.62" port: 45603 } }
I20260620 06:11:13.381682 1190 raft_consensus.cc:374] T 00000000000000000000000000000000 P 70868dce29ee4f89ace46943797eda44 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 06:11:13.381834 1190 raft_consensus.cc:729] T 00000000000000000000000000000000 P 70868dce29ee4f89ace46943797eda44 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 70868dce29ee4f89ace46943797eda44, State: Initialized, Role: FOLLOWER
I20260620 06:11:13.382334 1190 consensus_queue.cc:260] T 00000000000000000000000000000000 P 70868dce29ee4f89ace46943797eda44 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70868dce29ee4f89ace46943797eda44" member_type: VOTER last_known_addr { host: "127.11.102.62" port: 45603 } }
I20260620 06:11:13.382493 1190 raft_consensus.cc:388] T 00000000000000000000000000000000 P 70868dce29ee4f89ace46943797eda44 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 06:11:13.382582 1190 raft_consensus.cc:482] T 00000000000000000000000000000000 P 70868dce29ee4f89ace46943797eda44 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 06:11:13.382699 1190 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 70868dce29ee4f89ace46943797eda44 [term 0 FOLLOWER]: Advancing to term 1
I20260620 06:11:13.382773 11672 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 06:11:13.383911 1190 raft_consensus.cc:504] T 00000000000000000000000000000000 P 70868dce29ee4f89ace46943797eda44 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70868dce29ee4f89ace46943797eda44" member_type: VOTER last_known_addr { host: "127.11.102.62" port: 45603 } }
I20260620 06:11:13.384307 1190 leader_election.cc:302] T 00000000000000000000000000000000 P 70868dce29ee4f89ace46943797eda44 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 70868dce29ee4f89ace46943797eda44; no voters:
I20260620 06:11:13.384625 1190 leader_election.cc:288] T 00000000000000000000000000000000 P 70868dce29ee4f89ace46943797eda44 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 06:11:13.385221 1199 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 70868dce29ee4f89ace46943797eda44 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 06:11:13.385464 1199 raft_consensus.cc:686] T 00000000000000000000000000000000 P 70868dce29ee4f89ace46943797eda44 [term 1 LEADER]: Becoming Leader. State: Replica: 70868dce29ee4f89ace46943797eda44, State: Running, Role: LEADER
I20260620 06:11:13.385783 1199 consensus_queue.cc:237] T 00000000000000000000000000000000 P 70868dce29ee4f89ace46943797eda44 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70868dce29ee4f89ace46943797eda44" member_type: VOTER last_known_addr { host: "127.11.102.62" port: 45603 } }
I20260620 06:11:13.386013 1190 sys_catalog.cc:564] T 00000000000000000000000000000000 P 70868dce29ee4f89ace46943797eda44 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 06:11:13.386927 1199 sys_catalog.cc:455] T 00000000000000000000000000000000 P 70868dce29ee4f89ace46943797eda44 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 70868dce29ee4f89ace46943797eda44. Latest consensus state: current_term: 1 leader_uuid: "70868dce29ee4f89ace46943797eda44" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70868dce29ee4f89ace46943797eda44" member_type: VOTER last_known_addr { host: "127.11.102.62" port: 45603 } } }
I20260620 06:11:13.387096 1199 sys_catalog.cc:458] T 00000000000000000000000000000000 P 70868dce29ee4f89ace46943797eda44 [sys.catalog]: This master's current role is: LEADER
I20260620 06:11:13.387463 1209 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 06:11:13.389171 1200 sys_catalog.cc:455] T 00000000000000000000000000000000 P 70868dce29ee4f89ace46943797eda44 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "70868dce29ee4f89ace46943797eda44" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70868dce29ee4f89ace46943797eda44" member_type: VOTER last_known_addr { host: "127.11.102.62" port: 45603 } } }
I20260620 06:11:13.389281 1200 sys_catalog.cc:458] T 00000000000000000000000000000000 P 70868dce29ee4f89ace46943797eda44 [sys.catalog]: This master's current role is: LEADER
I20260620 06:11:13.390846 1209 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 06:11:13.394986 1193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:11:13.385040 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35291 (local address 127.11.102.62:45603)
0620 06:11:13.385154 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0620 06:11:13.385158 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 06:11:13.385465 (+ 307us) server_negotiation.cc:407] Connection header received
0620 06:11:13.385617 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:11:13.385621 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:11:13.385668 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:11:13.385750 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:11:13.386610 (+ 860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:11:13.387425 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:11:13.388528 (+ 1103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:11:13.388851 (+ 323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:11:13.388945 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:11:13.391460 (+ 2515us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:11:13.391465 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:11:13.391467 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:11:13.391492 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:11:13.393205 (+ 1713us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:11:13.393649 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:11:13.393653 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:11:13.393655 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:11:13.393740 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:11:13.394025 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:11:13.394029 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:11:13.394031 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:11:13.394215 (+ 184us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:11:13.394371 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0620 06:11:13.394594 (+ 223us) server_negotiation.cc:299] Negotiation successful
0620 06:11:13.394711 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260620 06:11:13.397274 1209 catalog_manager.cc:1347] Generated new cluster ID: a8212e7f69214b6e9c19d93502bf58da
I20260620 06:11:13.397337 1209 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 06:11:13.420179 1209 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 06:11:13.421187 1209 catalog_manager.cc:1504] Loading token signing keys...
I20260620 06:11:13.428718 1209 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 70868dce29ee4f89ace46943797eda44: Generated new TSK 0
I20260620 06:11:13.429438 1209 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 06:11:13.441871 11672 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9HMp43/build/debug/bin/kudu with pid 1122
2026-06-20T06:11:13Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (1431 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781935603120653-11672-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 06:11:13 dist-test-slave-9hm5 krb5kdc[1221](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 06:11:13 dist-test-slave-9hm5 krb5kdc[1221](info): set up 2 sockets
Jun 20 06:11:13 dist-test-slave-9hm5 krb5kdc[1221](info): commencing operation
krb5kdc: starting...
W20260620 06:11:14.483965 11672 mini_kdc.cc:121] Time spent starting KDC: real 1.022s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781935603120653-11672-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781935603120653-11672-0/krb5kdc/test-user.keytab.
Jun 20 06:11:14 dist-test-slave-9hm5 krb5kdc[1221](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935874, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T06:11:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T06:11:14Z Disabled control of system clock
WARNING: no policy specified for kudu/127.11.102.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.102.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.102.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781935603120653-11672-0/krb5kdc/kudu_127.11.102.62.keytab.
Entry for principal kudu/127.11.102.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781935603120653-11672-0/krb5kdc/kudu_127.11.102.62.keytab.
I20260620 06:11:14.606180 11672 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9HMp43/build/debug/bin/kudu
/tmp/dist-test-task9HMp43/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781935603120653-11672-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781935603120653-11672-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781935603120653-11672-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781935603120653-11672-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.102.62:38613
--webserver_interface=127.11.102.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781935603120653-11672-0/krb5kdc/kudu_127.11.102.62.keytab
--principal=kudu/127.11.102.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.102.20:36057
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.102.62:38613
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781935603120653-11672-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781935603120653-11672-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781935603120653-11672-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 06:11:14.716262 1234 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:11:14.716542 1234 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:11:14.716584 1234 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:11:14.720695 1234 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 06:11:14.720759 1234 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:11:14.720778 1234 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:11:14.720795 1234 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 06:11:14.720810 1234 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 06:11:14.726306 1234 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.102.20:36057
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781935603120653-11672-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781935603120653-11672-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.102.62:38613
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781935603120653-11672-0/krb5kdc/kudu_127.11.102.62.keytab
--principal=kudu/127.11.102.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.102.62:38613
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781935603120653-11672-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.102.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1234
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781935603120653-11672-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 05:28:48 UTC on ccycloud.quasar-fixhms.root.comops.site
build id 4457020
I20260620 06:11:14.727413 1234 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:11:14.728462 1234 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 06:11:14.736802 1239 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:11:14.736966 1234 server_base.cc:1034] running on GCE node
W20260620 06:11:14.736840 1242 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:11:14.736862 1240 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:11:14.737735 1234 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:11:14.738940 1234 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:11:14.740125 1234 hybrid_clock.cc:648] HybridClock initialized: now 1781935874740096 us; error 41 us; skew 500 ppm
Jun 20 06:11:14 dist-test-slave-9hm5 krb5kdc[1221](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935874, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.102.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 06:11:14.742964 1234 init.cc:377] Logged in from keytab as kudu/127.11.102.62@KRBTEST.COM (short username kudu)
I20260620 06:11:14.743970 1234 webserver.cc:460] Webserver started at http://127.11.102.62:38589/ using document root <none> and password file <none>
I20260620 06:11:14.744540 1234 fs_manager.cc:359] Metadata directory not provided
I20260620 06:11:14.744643 1234 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:11:14.744871 1234 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 06:11:14.746917 1234 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781935603120653-11672-0/minicluster-data/master-0/data/instance:
uuid: "42c229f38c324f958a9f68f242bc4ccf"
format_stamp: "Formatted at 2026-06-20 06:11:14 on dist-test-slave-9hm5"
server_key: "472834d1bfd3374b30683050bd6f2624"
server_key_iv: "3909c978e1c5c6c0430842f3c5bb0961"
server_key_version: "encryptionkey@0"
I20260620 06:11:14.747480 1234 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781935603120653-11672-0/minicluster-data/master-0/wal/instance:
uuid: "42c229f38c324f958a9f68f242bc4ccf"
format_stamp: "Formatted at 2026-06-20 06:11:14 on dist-test-slave-9hm5"
server_key: "472834d1bfd3374b30683050bd6f2624"
server_key_iv: "3909c978e1c5c6c0430842f3c5bb0961"
server_key_version: "encryptionkey@0"
I20260620 06:11:14.751225 1234 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260620 06:11:14.753786 1249 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:11:14.755108 1234 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260620 06:11:14.755272 1234 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781935603120653-11672-0/minicluster-data/master-0/data,/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781935603120653-11672-0/minicluster-data/master-0/wal
uuid: "42c229f38c324f958a9f68f242bc4ccf"
format_stamp: "Formatted at 2026-06-20 06:11:14 on dist-test-slave-9hm5"
server_key: "472834d1bfd3374b30683050bd6f2624"
server_key_iv: "3909c978e1c5c6c0430842f3c5bb0961"
server_key_version: "encryptionkey@0"
I20260620 06:11:14.755398 1234 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781935603120653-11672-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781935603120653-11672-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781935603120653-11672-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:11:14.768519 1234 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:11:14.771973 1234 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:11:14.772185 1234 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:11:14.779800 1234 rpc_server.cc:307] RPC server started. Bound to: 127.11.102.62:38613
I20260620 06:11:14.779876 1301 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.102.62:38613 every 8 connection(s)
I20260620 06:11:14.780827 1234 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781935603120653-11672-0/minicluster-data/master-0/data/info.pb
I20260620 06:11:14.782521 11672 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9HMp43/build/debug/bin/kudu as pid 1234
I20260620 06:11:14.782665 11672 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781935603120653-11672-0/minicluster-data/master-0/wal/instance
I20260620 06:11:14.782964 11672 external_mini_cluster.cc:1428] Setting key 6d021efb95f91d611a421a7a97450c0e
I20260620 06:11:14.784008 1302 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 06:11:14.789489 1302 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 42c229f38c324f958a9f68f242bc4ccf: Bootstrap starting.
I20260620 06:11:14.791792 1302 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 42c229f38c324f958a9f68f242bc4ccf: Neither blocks nor log segments found. Creating new log.
I20260620 06:11:14.791864 1305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:11:14.784367 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49777 (local address 127.11.102.62:38613)
0620 06:11:14.784784 (+ 417us) server_negotiation.cc:206] Beginning negotiation
0620 06:11:14.784799 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 06:11:14.784833 (+ 34us) server_negotiation.cc:407] Connection header received
0620 06:11:14.785484 (+ 651us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:11:14.785500 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:11:14.785831 (+ 331us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:11:14.786175 (+ 344us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:11:14.786583 (+ 408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:11:14.787567 (+ 984us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:11:14.788520 (+ 953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:11:14.789004 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:11:14.789196 (+ 192us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:11:14.790111 (+ 915us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:11:14.790117 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:11:14.790136 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 06:11:14.790519 (+ 383us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:11:14.790572 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 06:11:14.790594 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:11:14.790706 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0620 06:11:14.790846 (+ 140us) server_negotiation.cc:299] Negotiation successful
0620 06:11:14.791061 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":284,"thread_start_us":138,"threads_started":1}
I20260620 06:11:14.792542 1302 log.cc:824] T 00000000000000000000000000000000 P 42c229f38c324f958a9f68f242bc4ccf: Log is configured to *not* fsync() on all Append() calls
I20260620 06:11:14.794762 1302 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 42c229f38c324f958a9f68f242bc4ccf: No bootstrap required, opened a new log
I20260620 06:11:14.794757 11672 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 06:11:14.798928 1302 raft_consensus.cc:348] T 00000000000000000000000000000000 P 42c229f38c324f958a9f68f242bc4ccf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "42c229f38c324f958a9f68f242bc4ccf" member_type: VOTER last_known_addr { host: "127.11.102.62" port: 38613 } }
I20260620 06:11:14.799239 1302 raft_consensus.cc:374] T 00000000000000000000000000000000 P 42c229f38c324f958a9f68f242bc4ccf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 06:11:14.799304 1302 raft_consensus.cc:729] T 00000000000000000000000000000000 P 42c229f38c324f958a9f68f242bc4ccf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 42c229f38c324f958a9f68f242bc4ccf, State: Initialized, Role: FOLLOWER
I20260620 06:11:14.799804 1302 consensus_queue.cc:260] T 00000000000000000000000000000000 P 42c229f38c324f958a9f68f242bc4ccf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "42c229f38c324f958a9f68f242bc4ccf" member_type: VOTER last_known_addr { host: "127.11.102.62" port: 38613 } }
I20260620 06:11:14.800014 1302 raft_consensus.cc:388] T 00000000000000000000000000000000 P 42c229f38c324f958a9f68f242bc4ccf [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 06:11:14.800083 1302 raft_consensus.cc:482] T 00000000000000000000000000000000 P 42c229f38c324f958a9f68f242bc4ccf [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 06:11:14.800199 1302 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 42c229f38c324f958a9f68f242bc4ccf [term 0 FOLLOWER]: Advancing to term 1
I20260620 06:11:14.801011 1305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:11:14.796796 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38309 (local address 127.11.102.62:38613)
0620 06:11:14.796917 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0620 06:11:14.796920 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 06:11:14.797043 (+ 123us) server_negotiation.cc:407] Connection header received
0620 06:11:14.797184 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:11:14.797187 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:11:14.797224 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:11:14.797282 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:11:14.797618 (+ 336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:11:14.798118 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:11:14.799208 (+ 1090us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:11:14.799547 (+ 339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:11:14.799686 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:11:14.800572 (+ 886us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:11:14.800575 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:11:14.800577 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 06:11:14.800588 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:11:14.800612 (+ 24us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 06:11:14.800622 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:11:14.800682 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0620 06:11:14.800801 (+ 119us) server_negotiation.cc:299] Negotiation successful
0620 06:11:14.800852 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260620 06:11:14.801447 1302 raft_consensus.cc:504] T 00000000000000000000000000000000 P 42c229f38c324f958a9f68f242bc4ccf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "42c229f38c324f958a9f68f242bc4ccf" member_type: VOTER last_known_addr { host: "127.11.102.62" port: 38613 } }
I20260620 06:11:14.801787 1302 leader_election.cc:302] T 00000000000000000000000000000000 P 42c229f38c324f958a9f68f242bc4ccf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 42c229f38c324f958a9f68f242bc4ccf; no voters:
I20260620 06:11:14.802142 1302 leader_election.cc:288] T 00000000000000000000000000000000 P 42c229f38c324f958a9f68f242bc4ccf [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 06:11:14.802294 1312 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 42c229f38c324f958a9f68f242bc4ccf [term 1 FOLLOWER]: Leader election won for term 1
I20260620 06:11:14.802527 1312 raft_consensus.cc:686] T 00000000000000000000000000000000 P 42c229f38c324f958a9f68f242bc4ccf [term 1 LEADER]: Becoming Leader. State: Replica: 42c229f38c324f958a9f68f242bc4ccf, State: Running, Role: LEADER
I20260620 06:11:14.802879 1312 consensus_queue.cc:237] T 00000000000000000000000000000000 P 42c229f38c324f958a9f68f242bc4ccf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "42c229f38c324f958a9f68f242bc4ccf" member_type: VOTER last_known_addr { host: "127.11.102.62" port: 38613 } }
I20260620 06:11:14.803352 1302 sys_catalog.cc:564] T 00000000000000000000000000000000 P 42c229f38c324f958a9f68f242bc4ccf [sys.catalog]: configured and running, proceeding with master startup.
I20260620 06:11:14.804755 1314 sys_catalog.cc:455] T 00000000000000000000000000000000 P 42c229f38c324f958a9f68f242bc4ccf [sys.catalog]: SysCatalogTable state changed. Reason: New leader 42c229f38c324f958a9f68f242bc4ccf. Latest consensus state: current_term: 1 leader_uuid: "42c229f38c324f958a9f68f242bc4ccf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "42c229f38c324f958a9f68f242bc4ccf" member_type: VOTER last_known_addr { host: "127.11.102.62" port: 38613 } } }
I20260620 06:11:14.804791 1313 sys_catalog.cc:455] T 00000000000000000000000000000000 P 42c229f38c324f958a9f68f242bc4ccf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "42c229f38c324f958a9f68f242bc4ccf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "42c229f38c324f958a9f68f242bc4ccf" member_type: VOTER last_known_addr { host: "127.11.102.62" port: 38613 } } }
I20260620 06:11:14.804873 1314 sys_catalog.cc:458] T 00000000000000000000000000000000 P 42c229f38c324f958a9f68f242bc4ccf [sys.catalog]: This master's current role is: LEADER
I20260620 06:11:14.804884 1313 sys_catalog.cc:458] T 00000000000000000000000000000000 P 42c229f38c324f958a9f68f242bc4ccf [sys.catalog]: This master's current role is: LEADER
I20260620 06:11:14.805274 1322 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 06:11:14.808075 1322 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 06:11:14.814267 1322 catalog_manager.cc:1347] Generated new cluster ID: 231827c60c364a5d87feacc3034155b0
I20260620 06:11:14.814342 1322 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 06:11:14.821806 1322 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 06:11:14.823066 1322 catalog_manager.cc:1504] Loading token signing keys...
I20260620 06:11:14.829072 1322 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 42c229f38c324f958a9f68f242bc4ccf: Generated new TSK 0
I20260620 06:11:14.829890 1322 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 06:11:14.837992 11672 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9HMp43/build/debug/bin/kudu with pid 1234
2026-06-20T06:11:14Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1396 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781935603120653-11672-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 06:11:14 dist-test-slave-9hm5 krb5kdc[1333](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 06:11:14 dist-test-slave-9hm5 krb5kdc[1333](info): set up 2 sockets
Jun 20 06:11:14 dist-test-slave-9hm5 krb5kdc[1333](info): commencing operation
krb5kdc: starting...
W20260620 06:11:15.868002 11672 mini_kdc.cc:121] Time spent starting KDC: real 1.010s user 0.005s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781935603120653-11672-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781935603120653-11672-0/krb5kdc/test-user.keytab.
Jun 20 06:11:15 dist-test-slave-9hm5 krb5kdc[1333](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935875, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T06:11:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T06:11:15Z Disabled control of system clock
WARNING: no policy specified for kudu/127.11.102.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.102.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.102.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781935603120653-11672-0/krb5kdc/kudu_127.11.102.62.keytab.
Entry for principal kudu/127.11.102.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781935603120653-11672-0/krb5kdc/kudu_127.11.102.62.keytab.
I20260620 06:11:15.987368 11672 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9HMp43/build/debug/bin/kudu
/tmp/dist-test-task9HMp43/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781935603120653-11672-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781935603120653-11672-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781935603120653-11672-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781935603120653-11672-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.102.62:41533
--webserver_interface=127.11.102.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781935603120653-11672-0/krb5kdc/kudu_127.11.102.62.keytab
--principal=kudu/127.11.102.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.102.20:39601
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.102.62:41533
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781935603120653-11672-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781935603120653-11672-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781935603120653-11672-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 06:11:16.097036 1346 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:11:16.097324 1346 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:11:16.097419 1346 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:11:16.101440 1346 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 06:11:16.101527 1346 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:11:16.101565 1346 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:11:16.101620 1346 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 06:11:16.101656 1346 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 06:11:16.106854 1346 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.102.20:39601
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781935603120653-11672-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781935603120653-11672-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.102.62:41533
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781935603120653-11672-0/krb5kdc/kudu_127.11.102.62.keytab
--principal=kudu/127.11.102.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.102.62:41533
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781935603120653-11672-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.102.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1346
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781935603120653-11672-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 05:28:48 UTC on ccycloud.quasar-fixhms.root.comops.site
build id 4457020
I20260620 06:11:16.108006 1346 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:11:16.109030 1346 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 06:11:16.117856 1354 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:11:16.117856 1352 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:11:16.117971 1351 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:11:16.118245 1346 server_base.cc:1034] running on GCE node
I20260620 06:11:16.118829 1346 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:11:16.119976 1346 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:11:16.121176 1346 hybrid_clock.cc:648] HybridClock initialized: now 1781935876121159 us; error 28 us; skew 500 ppm
Jun 20 06:11:16 dist-test-slave-9hm5 krb5kdc[1333](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935876, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.102.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 06:11:16.124009 1346 init.cc:377] Logged in from keytab as kudu/127.11.102.62@KRBTEST.COM (short username kudu)
I20260620 06:11:16.124918 1346 webserver.cc:460] Webserver started at http://127.11.102.62:36055/ using document root <none> and password file <none>
I20260620 06:11:16.125476 1346 fs_manager.cc:359] Metadata directory not provided
I20260620 06:11:16.125567 1346 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:11:16.125794 1346 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 06:11:16.127861 1346 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781935603120653-11672-0/minicluster-data/master-0/data/instance:
uuid: "e66ac6aafa4948c6906cd94e191b8658"
format_stamp: "Formatted at 2026-06-20 06:11:16 on dist-test-slave-9hm5"
server_key: "6f5c68bb77b8105bc9431a0084d3b3eb"
server_key_iv: "81c121d06f292b47d69e9a1da509d036"
server_key_version: "encryptionkey@0"
I20260620 06:11:16.128415 1346 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781935603120653-11672-0/minicluster-data/master-0/wal/instance:
uuid: "e66ac6aafa4948c6906cd94e191b8658"
format_stamp: "Formatted at 2026-06-20 06:11:16 on dist-test-slave-9hm5"
server_key: "6f5c68bb77b8105bc9431a0084d3b3eb"
server_key_iv: "81c121d06f292b47d69e9a1da509d036"
server_key_version: "encryptionkey@0"
I20260620 06:11:16.132130 1346 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260620 06:11:16.134549 1361 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:11:16.135852 1346 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 06:11:16.135982 1346 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781935603120653-11672-0/minicluster-data/master-0/data,/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781935603120653-11672-0/minicluster-data/master-0/wal
uuid: "e66ac6aafa4948c6906cd94e191b8658"
format_stamp: "Formatted at 2026-06-20 06:11:16 on dist-test-slave-9hm5"
server_key: "6f5c68bb77b8105bc9431a0084d3b3eb"
server_key_iv: "81c121d06f292b47d69e9a1da509d036"
server_key_version: "encryptionkey@0"
I20260620 06:11:16.136089 1346 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781935603120653-11672-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781935603120653-11672-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781935603120653-11672-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:11:16.140539 1346 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:11:16.143729 1346 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:11:16.143910 1346 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:11:16.150576 1346 rpc_server.cc:307] RPC server started. Bound to: 127.11.102.62:41533
I20260620 06:11:16.150599 1413 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.102.62:41533 every 8 connection(s)
I20260620 06:11:16.151494 1346 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781935603120653-11672-0/minicluster-data/master-0/data/info.pb
I20260620 06:11:16.153155 11672 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9HMp43/build/debug/bin/kudu as pid 1346
I20260620 06:11:16.153288 11672 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781935603120653-11672-0/minicluster-data/master-0/wal/instance
I20260620 06:11:16.153626 11672 external_mini_cluster.cc:1428] Setting key 457642915d923a71e369302aaef999c1
I20260620 06:11:16.154402 1414 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 06:11:16.158557 1417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:11:16.154969 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55747 (local address 127.11.102.62:41533)
0620 06:11:16.155403 (+ 434us) server_negotiation.cc:206] Beginning negotiation
0620 06:11:16.155417 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 06:11:16.155449 (+ 32us) server_negotiation.cc:407] Connection header received
0620 06:11:16.156143 (+ 694us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:11:16.156158 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:11:16.156429 (+ 271us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:11:16.156768 (+ 339us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:11:16.157474 (+ 706us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:11:16.157479 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:11:16.157490 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 06:11:16.157499 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:11:16.157546 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 06:11:16.157564 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:11:16.157706 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0620 06:11:16.157775 (+ 69us) server_negotiation.cc:299] Negotiation successful
0620 06:11:16.157923 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":305,"thread_start_us":150,"threads_started":1}
I20260620 06:11:16.160521 1414 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e66ac6aafa4948c6906cd94e191b8658: Bootstrap starting.
I20260620 06:11:16.161361 11672 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 06:11:16.162958 1414 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e66ac6aafa4948c6906cd94e191b8658: Neither blocks nor log segments found. Creating new log.
I20260620 06:11:16.163889 1414 log.cc:824] T 00000000000000000000000000000000 P e66ac6aafa4948c6906cd94e191b8658: Log is configured to *not* fsync() on all Append() calls
I20260620 06:11:16.164988 1417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:11:16.163332 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59227 (local address 127.11.102.62:41533)
0620 06:11:16.163459 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0620 06:11:16.163463 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 06:11:16.163663 (+ 200us) server_negotiation.cc:407] Connection header received
0620 06:11:16.163823 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:11:16.163825 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:11:16.163855 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:11:16.163918 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:11:16.164596 (+ 678us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:11:16.164601 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:11:16.164603 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 06:11:16.164614 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:11:16.164643 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 06:11:16.164660 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:11:16.164767 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0620 06:11:16.164836 (+ 69us) server_negotiation.cc:299] Negotiation successful
0620 06:11:16.164865 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260620 06:11:16.166185 1414 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e66ac6aafa4948c6906cd94e191b8658: No bootstrap required, opened a new log
I20260620 06:11:16.169742 1414 raft_consensus.cc:348] T 00000000000000000000000000000000 P e66ac6aafa4948c6906cd94e191b8658 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e66ac6aafa4948c6906cd94e191b8658" member_type: VOTER last_known_addr { host: "127.11.102.62" port: 41533 } }
I20260620 06:11:16.169955 1414 raft_consensus.cc:374] T 00000000000000000000000000000000 P e66ac6aafa4948c6906cd94e191b8658 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 06:11:16.169989 1414 raft_consensus.cc:729] T 00000000000000000000000000000000 P e66ac6aafa4948c6906cd94e191b8658 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e66ac6aafa4948c6906cd94e191b8658, State: Initialized, Role: FOLLOWER
I20260620 06:11:16.170348 1414 consensus_queue.cc:260] T 00000000000000000000000000000000 P e66ac6aafa4948c6906cd94e191b8658 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e66ac6aafa4948c6906cd94e191b8658" member_type: VOTER last_known_addr { host: "127.11.102.62" port: 41533 } }
I20260620 06:11:16.170455 1414 raft_consensus.cc:388] T 00000000000000000000000000000000 P e66ac6aafa4948c6906cd94e191b8658 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 06:11:16.170506 1414 raft_consensus.cc:482] T 00000000000000000000000000000000 P e66ac6aafa4948c6906cd94e191b8658 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 06:11:16.170574 1414 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e66ac6aafa4948c6906cd94e191b8658 [term 0 FOLLOWER]: Advancing to term 1
I20260620 06:11:16.171392 1414 raft_consensus.cc:504] T 00000000000000000000000000000000 P e66ac6aafa4948c6906cd94e191b8658 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e66ac6aafa4948c6906cd94e191b8658" member_type: VOTER last_known_addr { host: "127.11.102.62" port: 41533 } }
I20260620 06:11:16.171751 1414 leader_election.cc:302] T 00000000000000000000000000000000 P e66ac6aafa4948c6906cd94e191b8658 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e66ac6aafa4948c6906cd94e191b8658; no voters:
I20260620 06:11:16.172080 1414 leader_election.cc:288] T 00000000000000000000000000000000 P e66ac6aafa4948c6906cd94e191b8658 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 06:11:16.172144 1424 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e66ac6aafa4948c6906cd94e191b8658 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 06:11:16.172343 1424 raft_consensus.cc:686] T 00000000000000000000000000000000 P e66ac6aafa4948c6906cd94e191b8658 [term 1 LEADER]: Becoming Leader. State: Replica: e66ac6aafa4948c6906cd94e191b8658, State: Running, Role: LEADER
I20260620 06:11:16.172699 1424 consensus_queue.cc:237] T 00000000000000000000000000000000 P e66ac6aafa4948c6906cd94e191b8658 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e66ac6aafa4948c6906cd94e191b8658" member_type: VOTER last_known_addr { host: "127.11.102.62" port: 41533 } }
I20260620 06:11:16.173357 1414 sys_catalog.cc:564] T 00000000000000000000000000000000 P e66ac6aafa4948c6906cd94e191b8658 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 06:11:16.173913 1424 sys_catalog.cc:455] T 00000000000000000000000000000000 P e66ac6aafa4948c6906cd94e191b8658 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e66ac6aafa4948c6906cd94e191b8658. Latest consensus state: current_term: 1 leader_uuid: "e66ac6aafa4948c6906cd94e191b8658" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e66ac6aafa4948c6906cd94e191b8658" member_type: VOTER last_known_addr { host: "127.11.102.62" port: 41533 } } }
I20260620 06:11:16.174093 1424 sys_catalog.cc:458] T 00000000000000000000000000000000 P e66ac6aafa4948c6906cd94e191b8658 [sys.catalog]: This master's current role is: LEADER
I20260620 06:11:16.174311 1425 sys_catalog.cc:455] T 00000000000000000000000000000000 P e66ac6aafa4948c6906cd94e191b8658 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e66ac6aafa4948c6906cd94e191b8658" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e66ac6aafa4948c6906cd94e191b8658" member_type: VOTER last_known_addr { host: "127.11.102.62" port: 41533 } } }
I20260620 06:11:16.174474 1425 sys_catalog.cc:458] T 00000000000000000000000000000000 P e66ac6aafa4948c6906cd94e191b8658 [sys.catalog]: This master's current role is: LEADER
I20260620 06:11:16.174530 1431 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 06:11:16.177639 1431 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 06:11:16.183307 1431 catalog_manager.cc:1347] Generated new cluster ID: 4c9da9b1ed214144a2ebc74f019e9492
I20260620 06:11:16.183382 1431 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 06:11:16.198047 1431 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 06:11:16.198406 1431 catalog_manager.cc:1504] Loading token signing keys...
I20260620 06:11:16.214711 1431 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e66ac6aafa4948c6906cd94e191b8658: Generated new TSK 0
I20260620 06:11:16.215487 1431 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 06:11:16.225781 11672 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9HMp43/build/debug/bin/kudu with pid 1346
2026-06-20T06:11:16Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1389 ms)
[----------] 18 tests from AuthTokenIssuingTest (25267 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781935603120653-11672-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 06:11:16 dist-test-slave-9hm5 krb5kdc[1445](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 06:11:16 dist-test-slave-9hm5 krb5kdc[1445](info): set up 2 sockets
Jun 20 06:11:16 dist-test-slave-9hm5 krb5kdc[1445](info): commencing operation
krb5kdc: starting...
W20260620 06:11:17.267959 11672 mini_kdc.cc:121] Time spent starting KDC: real 1.021s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781935603120653-11672-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781935603120653-11672-0/krb5kdc/test-user.keytab.
Jun 20 06:11:17 dist-test-slave-9hm5 krb5kdc[1445](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935877, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T06:11:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T06:11:17Z Disabled control of system clock
WARNING: no policy specified for kudu/127.11.102.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.102.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.102.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781935603120653-11672-0/krb5kdc/kudu_127.11.102.62.keytab.
Entry for principal kudu/127.11.102.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781935603120653-11672-0/krb5kdc/kudu_127.11.102.62.keytab.
I20260620 06:11:17.390375 11672 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9HMp43/build/debug/bin/kudu
/tmp/dist-test-task9HMp43/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781935603120653-11672-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781935603120653-11672-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781935603120653-11672-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781935603120653-11672-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.102.62:41463
--webserver_interface=127.11.102.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781935603120653-11672-0/krb5kdc/kudu_127.11.102.62.keytab
--principal=kudu/127.11.102.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.102.20:36351
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.102.62:41463,127.11.102.61:45917,127.11.102.60:44551
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781935603120653-11672-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781935603120653-11672-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781935603120653-11672-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 06:11:17.498693 1458 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260620 06:11:17.499025 1458 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:11:17.499109 1458 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:11:17.499187 1458 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:11:17.503189 1458 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 06:11:17.503301 1458 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:11:17.503338 1458 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:11:17.503379 1458 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 06:11:17.503415 1458 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 06:11:17.508730 1458 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.102.20:36351
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781935603120653-11672-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781935603120653-11672-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.102.62:41463,127.11.102.61:45917,127.11.102.60:44551
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781935603120653-11672-0/krb5kdc/kudu_127.11.102.62.keytab
--principal=kudu/127.11.102.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.102.62:41463
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781935603120653-11672-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.102.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1458
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781935603120653-11672-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 05:28:48 UTC on ccycloud.quasar-fixhms.root.comops.site
build id 4457020
I20260620 06:11:17.509866 1458 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:11:17.510942 1458 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 06:11:17.519433 1464 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:11:17.519477 1466 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:11:17.519471 1463 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:11:17.519670 1458 server_base.cc:1034] running on GCE node
I20260620 06:11:17.520344 1458 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:11:17.521353 1458 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:11:17.522506 1458 hybrid_clock.cc:648] HybridClock initialized: now 1781935877522495 us; error 25 us; skew 500 ppm
Jun 20 06:11:17 dist-test-slave-9hm5 krb5kdc[1445](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935877, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.102.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 06:11:17.525172 1458 init.cc:377] Logged in from keytab as kudu/127.11.102.62@KRBTEST.COM (short username kudu)
I20260620 06:11:17.526113 1458 webserver.cc:460] Webserver started at http://127.11.102.62:36617/ using document root <none> and password file <none>
I20260620 06:11:17.526675 1458 fs_manager.cc:359] Metadata directory not provided
I20260620 06:11:17.526767 1458 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:11:17.526998 1458 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 06:11:17.529105 1458 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781935603120653-11672-0/minicluster-data/master-0/data/instance:
uuid: "6ec5bfd83ab54c06920f04ce454a50b6"
format_stamp: "Formatted at 2026-06-20 06:11:17 on dist-test-slave-9hm5"
server_key: "ccdb769753e0b9202ae2be2f07e35916"
server_key_iv: "49e5f0e0df407941d380f561852230d8"
server_key_version: "encryptionkey@0"
I20260620 06:11:17.529675 1458 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781935603120653-11672-0/minicluster-data/master-0/wal/instance:
uuid: "6ec5bfd83ab54c06920f04ce454a50b6"
format_stamp: "Formatted at 2026-06-20 06:11:17 on dist-test-slave-9hm5"
server_key: "ccdb769753e0b9202ae2be2f07e35916"
server_key_iv: "49e5f0e0df407941d380f561852230d8"
server_key_version: "encryptionkey@0"
I20260620 06:11:17.533440 1458 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260620 06:11:17.535835 1473 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:11:17.536952 1458 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 06:11:17.537093 1458 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781935603120653-11672-0/minicluster-data/master-0/data,/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781935603120653-11672-0/minicluster-data/master-0/wal
uuid: "6ec5bfd83ab54c06920f04ce454a50b6"
format_stamp: "Formatted at 2026-06-20 06:11:17 on dist-test-slave-9hm5"
server_key: "ccdb769753e0b9202ae2be2f07e35916"
server_key_iv: "49e5f0e0df407941d380f561852230d8"
server_key_version: "encryptionkey@0"
I20260620 06:11:17.537201 1458 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781935603120653-11672-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781935603120653-11672-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781935603120653-11672-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:11:17.562951 1458 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:11:17.566150 1458 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:11:17.566377 1458 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:11:17.573575 1458 rpc_server.cc:307] RPC server started. Bound to: 127.11.102.62:41463
I20260620 06:11:17.573853 1525 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.102.62:41463 every 8 connection(s)
I20260620 06:11:17.574563 1458 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781935603120653-11672-0/minicluster-data/master-0/data/info.pb
I20260620 06:11:17.576452 11672 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9HMp43/build/debug/bin/kudu as pid 1458
I20260620 06:11:17.576601 11672 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781935603120653-11672-0/minicluster-data/master-0/wal/instance
I20260620 06:11:17.576901 11672 external_mini_cluster.cc:1428] Setting key e6f15cbd79ca930a00c894052dc9733c
I20260620 06:11:17.577498 1526 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 06:11:17.581630 1526 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.11.102.62" port: 41463 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.11.102.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.102.61@KRBTEST.COM" created.
Jun 20 06:11:17 dist-test-slave-9hm5 krb5kdc[1445](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935877, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.102.62@KRBTEST.COM for kudu/127.11.102.62@KRBTEST.COM
I20260620 06:11:17.596650 1530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:11:17.583953 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46302 (local address 127.11.102.62:41463)
0620 06:11:17.584543 (+ 590us) server_negotiation.cc:206] Beginning negotiation
0620 06:11:17.584557 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 06:11:17.584602 (+ 45us) server_negotiation.cc:407] Connection header received
0620 06:11:17.585638 (+ 1036us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:11:17.585643 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:11:17.585822 (+ 179us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:11:17.586007 (+ 185us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:11:17.586933 (+ 926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:11:17.587942 (+ 1009us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:11:17.588912 (+ 970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:11:17.589290 (+ 378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:11:17.589459 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:11:17.591807 (+ 2348us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:11:17.591812 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:11:17.591822 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:11:17.591841 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:11:17.593516 (+ 1675us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:11:17.594012 (+ 496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:11:17.594018 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:11:17.594020 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:11:17.594126 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:11:17.594439 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:11:17.594444 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:11:17.594446 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:11:17.594817 (+ 371us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:11:17.594960 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0620 06:11:17.595312 (+ 352us) server_negotiation.cc:299] Negotiation successful
0620 06:11:17.595557 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":453,"spinlock_wait_cycles":12672,"thread_start_us":150,"threads_started":1}
I20260620 06:11:17.596650 1529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:11:17.583953 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.102.62:41463 (local address 127.0.0.1:46302)
0620 06:11:17.584425 (+ 472us) negotiation.cc:107] Waiting for socket to connect
0620 06:11:17.584478 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0620 06:11:17.585316 (+ 838us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:11:17.586002 (+ 686us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:11:17.586006 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:11:17.586498 (+ 492us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:11:17.586783 (+ 285us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:11:17.586798 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:11:17.588065 (+ 1267us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:11:17.588068 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:11:17.588791 (+ 723us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:11:17.588800 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:11:17.589446 (+ 646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:11:17.589450 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:11:17.589560 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:11:17.590148 (+ 588us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:11:17.590164 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:11:17.591593 (+ 1429us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:11:17.593649 (+ 2056us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:11:17.593671 (+ 22us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:11:17.593682 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:11:17.593902 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:11:17.594226 (+ 324us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:11:17.594231 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:11:17.594233 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:11:17.594342 (+ 109us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:11:17.594928 (+ 586us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:11:17.594934 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:11:17.595073 (+ 139us) client_negotiation.cc:769] Sending connection context
0620 06:11:17.595238 (+ 165us) client_negotiation.cc:240] Negotiation successful
0620 06:11:17.595606 (+ 368us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":274,"mutex_wait_us":108,"thread_start_us":143,"threads_started":1}
I20260620 06:11:17.599166 1526 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.11.102.61" port: 45917 } has no permanent_uuid. Determining permanent_uuid...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.102.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781935603120653-11672-0/krb5kdc/kudu_127.11.102.61.keytab.
Entry for principal kudu/127.11.102.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781935603120653-11672-0/krb5kdc/kudu_127.11.102.61.keytab.
I20260620 06:11:17.600037 1529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:11:17.599693 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.102.61:45917 (local address 127.0.0.1:60302)
0620 06:11:17.599837 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0620 06:11:17.599942 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.102.61:45917: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260620 06:11:17.600253 1474 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.11.102.61:45917: connect: Connection refused (error 111)
I20260620 06:11:17.600754 11672 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9HMp43/build/debug/bin/kudu
/tmp/dist-test-task9HMp43/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781935603120653-11672-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781935603120653-11672-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781935603120653-11672-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781935603120653-11672-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.102.61:45917
--webserver_interface=127.11.102.61
--webserver_port=0
--keytab_file=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781935603120653-11672-0/krb5kdc/kudu_127.11.102.61.keytab
--principal=kudu/127.11.102.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.102.20:36351
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.102.62:41463,127.11.102.61:45917,127.11.102.60:44551
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781935603120653-11672-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781935603120653-11672-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781935603120653-11672-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 06:11:17.601155 1529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:11:17.600899 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.102.61:45917 (local address 127.0.0.1:60310)
0620 06:11:17.601067 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0620 06:11:17.601105 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.102.61:45917: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112}
W20260620 06:11:17.601320 1526 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.11.102.61:45917: Network error: Client connection negotiation failed: client connection to 127.11.102.61:45917: connect: Connection refused (error 111)
I20260620 06:11:17.650547 1526 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.11.102.61" port: 45917 } attempt: 1
I20260620 06:11:17.651473 1529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:11:17.651088 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.102.61:45917 (local address 127.0.0.1:60320)
0620 06:11:17.651263 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0620 06:11:17.651350 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.102.61:45917: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
I20260620 06:11:17.652272 1529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:11:17.652044 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.102.61:45917 (local address 127.0.0.1:60324)
0620 06:11:17.652151 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0620 06:11:17.652209 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.102.61:45917: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260620 06:11:17.652413 1526 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.11.102.61:45917: Network error: Client connection negotiation failed: client connection to 127.11.102.61:45917: connect: Connection refused (error 111)
W20260620 06:11:17.712786 1533 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260620 06:11:17.713063 1533 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:11:17.713086 1533 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:11:17.713124 1533 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:11:17.717307 1533 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 06:11:17.717366 1533 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:11:17.717386 1533 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:11:17.717402 1533 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 06:11:17.717417 1533 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 06:11:17.720566 1526 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.11.102.61" port: 45917 } attempt: 2
I20260620 06:11:17.721412 1529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:11:17.721090 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.102.61:45917 (local address 127.0.0.1:60326)
0620 06:11:17.721252 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0620 06:11:17.721323 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.102.61:45917: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260620 06:11:17.722137 1529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:11:17.721926 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.102.61:45917 (local address 127.0.0.1:60330)
0620 06:11:17.722036 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0620 06:11:17.722083 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.102.61:45917: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260620 06:11:17.722280 1526 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.11.102.61:45917: Network error: Client connection negotiation failed: client connection to 127.11.102.61:45917: connect: Connection refused (error 111)
I20260620 06:11:17.723210 1533 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.102.20:36351
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781935603120653-11672-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781935603120653-11672-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.11.102.62:41463,127.11.102.61:45917,127.11.102.60:44551
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781935603120653-11672-0/krb5kdc/kudu_127.11.102.61.keytab
--principal=kudu/127.11.102.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.102.61:45917
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781935603120653-11672-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.11.102.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1533
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781935603120653-11672-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 05:28:48 UTC on ccycloud.quasar-fixhms.root.comops.site
build id 4457020
I20260620 06:11:17.724344 1533 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:11:17.725358 1533 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 06:11:17.733553 1538 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:11:17.733577 1539 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:11:17.733580 1541 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:11:17.733776 1533 server_base.cc:1034] running on GCE node
I20260620 06:11:17.734361 1533 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:11:17.735350 1533 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:11:17.736506 1533 hybrid_clock.cc:648] HybridClock initialized: now 1781935877736470 us; error 47 us; skew 500 ppm
Jun 20 06:11:17 dist-test-slave-9hm5 krb5kdc[1445](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935877, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.102.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 06:11:17.739207 1533 init.cc:377] Logged in from keytab as kudu/127.11.102.61@KRBTEST.COM (short username kudu)
I20260620 06:11:17.740178 1533 webserver.cc:460] Webserver started at http://127.11.102.61:40453/ using document root <none> and password file <none>
I20260620 06:11:17.740782 1533 fs_manager.cc:359] Metadata directory not provided
I20260620 06:11:17.740846 1533 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:11:17.741071 1533 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 06:11:17.743110 1533 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781935603120653-11672-0/minicluster-data/master-1/data/instance:
uuid: "e15f6538c09e4081a3a9f37124026a96"
format_stamp: "Formatted at 2026-06-20 06:11:17 on dist-test-slave-9hm5"
server_key: "429df6fbc04816d24b4c76d265388a66"
server_key_iv: "e9d5d8e264b10aa8632d640348f6c334"
server_key_version: "encryptionkey@0"
I20260620 06:11:17.743726 1533 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781935603120653-11672-0/minicluster-data/master-1/wal/instance:
uuid: "e15f6538c09e4081a3a9f37124026a96"
format_stamp: "Formatted at 2026-06-20 06:11:17 on dist-test-slave-9hm5"
server_key: "429df6fbc04816d24b4c76d265388a66"
server_key_iv: "e9d5d8e264b10aa8632d640348f6c334"
server_key_version: "encryptionkey@0"
I20260620 06:11:17.747327 1533 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260620 06:11:17.749715 1548 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:11:17.750903 1533 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 06:11:17.751045 1533 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781935603120653-11672-0/minicluster-data/master-1/data,/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781935603120653-11672-0/minicluster-data/master-1/wal
uuid: "e15f6538c09e4081a3a9f37124026a96"
format_stamp: "Formatted at 2026-06-20 06:11:17 on dist-test-slave-9hm5"
server_key: "429df6fbc04816d24b4c76d265388a66"
server_key_iv: "e9d5d8e264b10aa8632d640348f6c334"
server_key_version: "encryptionkey@0"
I20260620 06:11:17.751157 1533 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781935603120653-11672-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781935603120653-11672-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781935603120653-11672-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:11:17.773079 1533 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:11:17.776542 1533 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:11:17.776772 1533 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:11:17.784157 1533 rpc_server.cc:307] RPC server started. Bound to: 127.11.102.61:45917
I20260620 06:11:17.784507 1600 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.102.61:45917 every 8 connection(s)
I20260620 06:11:17.785177 1533 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781935603120653-11672-0/minicluster-data/master-1/data/info.pb
I20260620 06:11:17.787067 11672 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9HMp43/build/debug/bin/kudu as pid 1533
I20260620 06:11:17.787209 11672 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781935603120653-11672-0/minicluster-data/master-1/wal/instance
I20260620 06:11:17.787492 11672 external_mini_cluster.cc:1428] Setting key 68b7dcd1ea623cf861665cf84f12a04c
I20260620 06:11:17.788118 1601 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 06:11:17.791898 1601 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.11.102.62" port: 41463 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.11.102.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.102.60@KRBTEST.COM" created.
Jun 20 06:11:17 dist-test-slave-9hm5 krb5kdc[1445](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935877, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.102.61@KRBTEST.COM for kudu/127.11.102.62@KRBTEST.COM
I20260620 06:11:17.805553 1530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:11:17.793879 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46310 (local address 127.11.102.62:41463)
0620 06:11:17.794010 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0620 06:11:17.794014 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 06:11:17.795143 (+ 1129us) server_negotiation.cc:407] Connection header received
0620 06:11:17.796164 (+ 1021us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:11:17.796168 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:11:17.796231 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:11:17.796321 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:11:17.797492 (+ 1171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:11:17.798060 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:11:17.799229 (+ 1169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:11:17.799476 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:11:17.799590 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:11:17.802032 (+ 2442us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:11:17.802035 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:11:17.802037 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:11:17.802060 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:11:17.803487 (+ 1427us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:11:17.804020 (+ 533us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:11:17.804028 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:11:17.804029 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:11:17.804103 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:11:17.804453 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:11:17.804456 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:11:17.804458 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:11:17.804633 (+ 175us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:11:17.804758 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0620 06:11:17.805253 (+ 495us) server_negotiation.cc:299] Negotiation successful
0620 06:11:17.805376 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260620 06:11:17.806344 1604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:11:17.794414 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.102.62:41463 (local address 127.0.0.1:46310)
0620 06:11:17.794946 (+ 532us) negotiation.cc:107] Waiting for socket to connect
0620 06:11:17.795027 (+ 81us) client_negotiation.cc:174] Beginning negotiation
0620 06:11:17.795951 (+ 924us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:11:17.796500 (+ 549us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:11:17.796507 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:11:17.796843 (+ 336us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:11:17.797339 (+ 496us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:11:17.797352 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:11:17.798194 (+ 842us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:11:17.798208 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:11:17.799092 (+ 884us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:11:17.799101 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:11:17.799655 (+ 554us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:11:17.799660 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:11:17.799835 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:11:17.800463 (+ 628us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:11:17.800479 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:11:17.801863 (+ 1384us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:11:17.803605 (+ 1742us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:11:17.803631 (+ 26us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:11:17.803646 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:11:17.803906 (+ 260us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:11:17.804208 (+ 302us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:11:17.804211 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:11:17.804213 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:11:17.804350 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:11:17.804756 (+ 406us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:11:17.804763 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:11:17.805006 (+ 243us) client_negotiation.cc:769] Sending connection context
0620 06:11:17.805186 (+ 180us) client_negotiation.cc:240] Negotiation successful
0620 06:11:17.805511 (+ 325us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":331,"thread_start_us":148,"threads_started":1}
I20260620 06:11:17.807876 1601 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.11.102.61" port: 45917 } has no permanent_uuid. Determining permanent_uuid...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.102.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781935603120653-11672-0/krb5kdc/kudu_127.11.102.60.keytab.
Entry for principal kudu/127.11.102.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781935603120653-11672-0/krb5kdc/kudu_127.11.102.60.keytab.
I20260620 06:11:17.812621 11672 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9HMp43/build/debug/bin/kudu
/tmp/dist-test-task9HMp43/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781935603120653-11672-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781935603120653-11672-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781935603120653-11672-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781935603120653-11672-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.102.60:44551
--webserver_interface=127.11.102.60
--webserver_port=0
--keytab_file=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781935603120653-11672-0/krb5kdc/kudu_127.11.102.60.keytab
--principal=kudu/127.11.102.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.102.20:36351
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.102.62:41463,127.11.102.61:45917,127.11.102.60:44551
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781935603120653-11672-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781935603120653-11672-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781935603120653-11672-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 06:11:17.813432 1526 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.11.102.61" port: 45917 } attempt: 3
Jun 20 06:11:17 dist-test-slave-9hm5 krb5kdc[1445](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935877, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.102.61@KRBTEST.COM for kudu/127.11.102.61@KRBTEST.COM
I20260620 06:11:17.818902 1604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:11:17.808498 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.102.61:45917 (local address 127.0.0.1:60346)
0620 06:11:17.808636 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0620 06:11:17.808651 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 06:11:17.808755 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:11:17.809514 (+ 759us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:11:17.809516 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:11:17.809646 (+ 130us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:11:17.809781 (+ 135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:11:17.809788 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:11:17.810805 (+ 1017us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:11:17.810807 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:11:17.811416 (+ 609us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:11:17.811429 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:11:17.812253 (+ 824us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:11:17.812257 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:11:17.812388 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:11:17.812893 (+ 505us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:11:17.812905 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:11:17.814281 (+ 1376us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:11:17.816667 (+ 2386us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:11:17.816673 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:11:17.816677 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:11:17.816943 (+ 266us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:11:17.817265 (+ 322us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:11:17.817268 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:11:17.817270 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:11:17.817365 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:11:17.818354 (+ 989us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:11:17.818358 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:11:17.818448 (+ 90us) client_negotiation.cc:769] Sending connection context
0620 06:11:17.818575 (+ 127us) client_negotiation.cc:240] Negotiation successful
0620 06:11:17.818732 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":55}
Jun 20 06:11:17 dist-test-slave-9hm5 krb5kdc[1445](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935877, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.102.62@KRBTEST.COM for kudu/127.11.102.61@KRBTEST.COM
I20260620 06:11:17.819391 1606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:11:17.808825 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60346 (local address 127.11.102.61:45917)
0620 06:11:17.809125 (+ 300us) server_negotiation.cc:206] Beginning negotiation
0620 06:11:17.809133 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 06:11:17.809152 (+ 19us) server_negotiation.cc:407] Connection header received
0620 06:11:17.809184 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:11:17.809188 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:11:17.809392 (+ 204us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:11:17.809498 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:11:17.809913 (+ 415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:11:17.810691 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:11:17.811556 (+ 865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:11:17.812136 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:11:17.812226 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:11:17.814427 (+ 2201us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:11:17.814438 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:11:17.814454 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:11:17.814481 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:11:17.816545 (+ 2064us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:11:17.817070 (+ 525us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:11:17.817079 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:11:17.817082 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:11:17.817168 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:11:17.817870 (+ 702us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:11:17.817875 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:11:17.817877 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:11:17.818215 (+ 338us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:11:17.818380 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0620 06:11:17.818688 (+ 308us) server_negotiation.cc:299] Negotiation successful
0620 06:11:17.818852 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":216,"thread_start_us":126,"threads_started":1}
I20260620 06:11:17.821499 1601 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.11.102.60" port: 44551 } has no permanent_uuid. Determining permanent_uuid...
I20260620 06:11:17.822156 1604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:11:17.821925 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.102.60:44551 (local address 127.0.0.1:58866)
0620 06:11:17.822027 (+ 102us) negotiation.cc:107] Waiting for socket to connect
0620 06:11:17.822105 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.102.60:44551: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260620 06:11:17.822393 1551 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.11.102.60:44551: connect: Connection refused (error 111)
I20260620 06:11:17.823190 1604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:11:17.822991 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.102.60:44551 (local address 127.0.0.1:58878)
0620 06:11:17.823091 (+ 100us) negotiation.cc:107] Waiting for socket to connect
0620 06:11:17.823134 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.102.60:44551: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260620 06:11:17.823403 1601 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.11.102.60:44551: Network error: Client connection negotiation failed: client connection to 127.11.102.60:44551: connect: Connection refused (error 111)
I20260620 06:11:17.823526 1529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:11:17.813861 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.102.61:45917 (local address 127.0.0.1:60348)
0620 06:11:17.814003 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0620 06:11:17.814020 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 06:11:17.814103 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:11:17.814628 (+ 525us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:11:17.814632 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:11:17.814826 (+ 194us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:11:17.814968 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:11:17.814979 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:11:17.815932 (+ 953us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:11:17.815935 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:11:17.816944 (+ 1009us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:11:17.816954 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:11:17.817490 (+ 536us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:11:17.817495 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:11:17.817650 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:11:17.818100 (+ 450us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:11:17.818113 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:11:17.819580 (+ 1467us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:11:17.821637 (+ 2057us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:11:17.821641 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:11:17.821643 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:11:17.821923 (+ 280us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:11:17.822288 (+ 365us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:11:17.822294 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:11:17.822297 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:11:17.822398 (+ 101us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:11:17.822902 (+ 504us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:11:17.822908 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:11:17.823030 (+ 122us) client_negotiation.cc:769] Sending connection context
0620 06:11:17.823126 (+ 96us) client_negotiation.cc:240] Negotiation successful
0620 06:11:17.823301 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":41}
I20260620 06:11:17.824198 1608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:11:17.814111 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60348 (local address 127.11.102.61:45917)
0620 06:11:17.814428 (+ 317us) server_negotiation.cc:206] Beginning negotiation
0620 06:11:17.814434 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 06:11:17.814444 (+ 10us) server_negotiation.cc:407] Connection header received
0620 06:11:17.814469 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:11:17.814470 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:11:17.814509 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:11:17.814604 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:11:17.815090 (+ 486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:11:17.815817 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:11:17.817060 (+ 1243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:11:17.817363 (+ 303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:11:17.817466 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:11:17.819760 (+ 2294us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:11:17.819763 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:11:17.819764 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:11:17.819784 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:11:17.821515 (+ 1731us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:11:17.822054 (+ 539us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:11:17.822060 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:11:17.822063 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:11:17.822155 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:11:17.822532 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:11:17.822537 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:11:17.822541 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:11:17.822772 (+ 231us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:11:17.823746 (+ 974us) server_negotiation.cc:1035] Waiting for connection context
0620 06:11:17.823833 (+ 87us) server_negotiation.cc:299] Negotiation successful
0620 06:11:17.823960 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":239,"thread_start_us":100,"threads_started":1}
I20260620 06:11:17.824884 1526 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.11.102.60" port: 44551 } has no permanent_uuid. Determining permanent_uuid...
I20260620 06:11:17.825546 1529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:11:17.825316 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.102.60:44551 (local address 127.0.0.1:58880)
0620 06:11:17.825432 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0620 06:11:17.825483 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.102.60:44551: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20260620 06:11:17.826175 1529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:11:17.825945 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.102.60:44551 (local address 127.0.0.1:58890)
0620 06:11:17.826079 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0620 06:11:17.826126 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.102.60:44551: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
W20260620 06:11:17.826329 1526 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.11.102.60:44551: Network error: Client connection negotiation failed: client connection to 127.11.102.60:44551: connect: Connection refused (error 111)
I20260620 06:11:17.857527 1526 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.11.102.60" port: 44551 } attempt: 1
I20260620 06:11:17.858510 1529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:11:17.858088 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.102.60:44551 (local address 127.0.0.1:58896)
0620 06:11:17.858347 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0620 06:11:17.858415 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.102.60:44551: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":126}
I20260620 06:11:17.859983 1529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:11:17.859715 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.102.60:44551 (local address 127.0.0.1:58902)
0620 06:11:17.859857 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0620 06:11:17.859920 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.102.60:44551: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83}
W20260620 06:11:17.860206 1526 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.11.102.60:44551: Network error: Client connection negotiation failed: client connection to 127.11.102.60:44551: connect: Connection refused (error 111)
I20260620 06:11:17.872769 1601 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.11.102.60" port: 44551 } attempt: 1
I20260620 06:11:17.873566 1604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:11:17.873302 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.102.60:44551 (local address 127.0.0.1:58912)
0620 06:11:17.873434 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0620 06:11:17.873501 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.102.60:44551: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260620 06:11:17.874262 1604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:11:17.874062 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.102.60:44551 (local address 127.0.0.1:58926)
0620 06:11:17.874163 (+ 101us) negotiation.cc:107] Waiting for socket to connect
0620 06:11:17.874210 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.102.60:44551: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260620 06:11:17.874379 1601 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.11.102.60:44551: Network error: Client connection negotiation failed: client connection to 127.11.102.60:44551: connect: Connection refused (error 111)
W20260620 06:11:17.925442 1607 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260620 06:11:17.925872 1607 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:11:17.925937 1607 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:11:17.926010 1607 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:11:17.930019 1607 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 06:11:17.930099 1607 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:11:17.930137 1607 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:11:17.930193 1607 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 06:11:17.930230 1607 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 06:11:17.935397 1526 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.11.102.60" port: 44551 } attempt: 2
I20260620 06:11:17.935583 1607 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.102.20:36351
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781935603120653-11672-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781935603120653-11672-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.11.102.62:41463,127.11.102.61:45917,127.11.102.60:44551
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781935603120653-11672-0/krb5kdc/kudu_127.11.102.60.keytab
--principal=kudu/127.11.102.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.102.60:44551
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781935603120653-11672-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.11.102.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1607
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781935603120653-11672-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 05:28:48 UTC on ccycloud.quasar-fixhms.root.comops.site
build id 4457020
I20260620 06:11:17.936230 1529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:11:17.935937 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.102.60:44551 (local address 127.0.0.1:58928)
0620 06:11:17.936079 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0620 06:11:17.936138 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.102.60:44551: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260620 06:11:17.936838 1607 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:11:17.937109 1529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:11:17.936861 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.102.60:44551 (local address 127.0.0.1:58934)
0620 06:11:17.937016 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0620 06:11:17.937060 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.102.60:44551: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
W20260620 06:11:17.937340 1526 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.11.102.60:44551: Network error: Client connection negotiation failed: client connection to 127.11.102.60:44551: connect: Connection refused (error 111)
I20260620 06:11:17.937943 1607 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 06:11:17.942495 1601 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.11.102.60" port: 44551 } attempt: 2
I20260620 06:11:17.943213 1604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:11:17.942954 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.102.60:44551 (local address 127.0.0.1:58946)
0620 06:11:17.943078 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0620 06:11:17.943133 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.102.60:44551: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260620 06:11:17.943962 1604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:11:17.943793 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.102.60:44551 (local address 127.0.0.1:58958)
0620 06:11:17.943870 (+ 77us) negotiation.cc:107] Waiting for socket to connect
0620 06:11:17.943913 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.102.60:44551: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":23}
W20260620 06:11:17.944160 1601 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.11.102.60:44551: Network error: Client connection negotiation failed: client connection to 127.11.102.60:44551: connect: Connection refused (error 111)
W20260620 06:11:17.946934 1615 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:11:17.947005 1607 server_base.cc:1034] running on GCE node
W20260620 06:11:17.946934 1614 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:11:17.946964 1617 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:11:17.947723 1607 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:11:17.948871 1607 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:11:17.950057 1607 hybrid_clock.cc:648] HybridClock initialized: now 1781935877950024 us; error 49 us; skew 500 ppm
Jun 20 06:11:17 dist-test-slave-9hm5 krb5kdc[1445](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935877, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.102.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 06:11:17.952833 1607 init.cc:377] Logged in from keytab as kudu/127.11.102.60@KRBTEST.COM (short username kudu)
I20260620 06:11:17.953709 1607 webserver.cc:460] Webserver started at http://127.11.102.60:38755/ using document root <none> and password file <none>
I20260620 06:11:17.954358 1607 fs_manager.cc:359] Metadata directory not provided
I20260620 06:11:17.954455 1607 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:11:17.954694 1607 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 06:11:17.956808 1607 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781935603120653-11672-0/minicluster-data/master-2/data/instance:
uuid: "76492d24230c4ec5a5d4d934e1bd7930"
format_stamp: "Formatted at 2026-06-20 06:11:17 on dist-test-slave-9hm5"
server_key: "892572728a4db94643e31a35c5c9f4ca"
server_key_iv: "8f6d77b5dc3939d98114f8ab19854363"
server_key_version: "encryptionkey@0"
I20260620 06:11:17.957360 1607 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781935603120653-11672-0/minicluster-data/master-2/wal/instance:
uuid: "76492d24230c4ec5a5d4d934e1bd7930"
format_stamp: "Formatted at 2026-06-20 06:11:17 on dist-test-slave-9hm5"
server_key: "892572728a4db94643e31a35c5c9f4ca"
server_key_iv: "8f6d77b5dc3939d98114f8ab19854363"
server_key_version: "encryptionkey@0"
I20260620 06:11:17.960979 1607 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 06:11:17.963299 1624 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:11:17.964529 1607 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 06:11:17.964669 1607 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781935603120653-11672-0/minicluster-data/master-2/data,/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781935603120653-11672-0/minicluster-data/master-2/wal
uuid: "76492d24230c4ec5a5d4d934e1bd7930"
format_stamp: "Formatted at 2026-06-20 06:11:17 on dist-test-slave-9hm5"
server_key: "892572728a4db94643e31a35c5c9f4ca"
server_key_iv: "8f6d77b5dc3939d98114f8ab19854363"
server_key_version: "encryptionkey@0"
I20260620 06:11:17.964785 1607 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781935603120653-11672-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781935603120653-11672-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781935603120653-11672-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:11:17.983492 1607 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:11:17.986378 1607 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:11:17.986588 1607 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:11:17.993625 1607 rpc_server.cc:307] RPC server started. Bound to: 127.11.102.60:44551
I20260620 06:11:17.993909 1676 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.102.60:44551 every 8 connection(s)
I20260620 06:11:17.994621 1607 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781935603120653-11672-0/minicluster-data/master-2/data/info.pb
I20260620 06:11:17.997331 1677 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 06:11:17.999481 11672 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9HMp43/build/debug/bin/kudu as pid 1607
I20260620 06:11:17.999647 11672 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781935603120653-11672-0/minicluster-data/master-2/wal/instance
I20260620 06:11:17.999960 11672 external_mini_cluster.cc:1428] Setting key a30f5858a067936c69c9301fefe3dee0
I20260620 06:11:18.001716 1677 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.11.102.62" port: 41463 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 06:11:18 dist-test-slave-9hm5 krb5kdc[1445](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935877, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.102.60@KRBTEST.COM for kudu/127.11.102.62@KRBTEST.COM
Jun 20 06:11:18 dist-test-slave-9hm5 krb5kdc[1445](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935877, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.102.60@KRBTEST.COM
I20260620 06:11:18.013653 1530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:11:18.003410 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46314 (local address 127.11.102.62:41463)
0620 06:11:18.003551 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0620 06:11:18.003556 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 06:11:18.003960 (+ 404us) server_negotiation.cc:407] Connection header received
0620 06:11:18.004127 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:11:18.004130 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:11:18.004182 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:11:18.004282 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:11:18.004955 (+ 673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:11:18.005724 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:11:18.006895 (+ 1171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:11:18.007119 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:11:18.007237 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:11:18.009577 (+ 2340us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:11:18.009581 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:11:18.009584 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:11:18.009620 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:11:18.011228 (+ 1608us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:11:18.011844 (+ 616us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:11:18.011847 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:11:18.011848 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:11:18.011925 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:11:18.012322 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:11:18.012326 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:11:18.012328 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:11:18.012592 (+ 264us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:11:18.012695 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0620 06:11:18.013328 (+ 633us) server_negotiation.cc:299] Negotiation successful
0620 06:11:18.013467 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260620 06:11:18.014634 1680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:11:18.001358 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58968 (local address 127.11.102.60:44551)
0620 06:11:18.001788 (+ 430us) server_negotiation.cc:206] Beginning negotiation
0620 06:11:18.001806 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0620 06:11:18.001841 (+ 35us) server_negotiation.cc:407] Connection header received
0620 06:11:18.002521 (+ 680us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:11:18.002539 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:11:18.002879 (+ 340us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:11:18.003205 (+ 326us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:11:18.004094 (+ 889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:11:18.005343 (+ 1249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:11:18.006599 (+ 1256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:11:18.006981 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:11:18.007154 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:11:18.010020 (+ 2866us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:11:18.010029 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:11:18.010044 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:11:18.010072 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:11:18.012025 (+ 1953us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:11:18.012465 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:11:18.012469 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:11:18.012470 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:11:18.012533 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:11:18.012826 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:11:18.012829 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:11:18.012830 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:11:18.013101 (+ 271us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:11:18.013324 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0620 06:11:18.013540 (+ 216us) server_negotiation.cc:299] Negotiation successful
0620 06:11:18.013654 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":290,"thread_start_us":131,"threads_started":1}
I20260620 06:11:18.014634 1681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:11:18.003383 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.102.62:41463 (local address 127.0.0.1:46314)
0620 06:11:18.003806 (+ 423us) negotiation.cc:107] Waiting for socket to connect
0620 06:11:18.003853 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0620 06:11:18.003975 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:11:18.004329 (+ 354us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:11:18.004333 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:11:18.004648 (+ 315us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:11:18.004818 (+ 170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:11:18.004828 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:11:18.005858 (+ 1030us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:11:18.005861 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:11:18.006762 (+ 901us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:11:18.006771 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:11:18.007267 (+ 496us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:11:18.007271 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:11:18.007394 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:11:18.007961 (+ 567us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:11:18.007976 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:11:18.009436 (+ 1460us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:11:18.011348 (+ 1912us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:11:18.011357 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:11:18.011371 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:11:18.011747 (+ 376us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:11:18.012018 (+ 271us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:11:18.012021 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:11:18.012022 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:11:18.012198 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:11:18.012698 (+ 500us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:11:18.012702 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:11:18.013019 (+ 317us) client_negotiation.cc:769] Sending connection context
0620 06:11:18.013216 (+ 197us) client_negotiation.cc:240] Negotiation successful
0620 06:11:18.013569 (+ 353us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":263,"spinlock_wait_cycles":768,"thread_start_us":132,"threads_started":1}
I20260620 06:11:18.015910 1677 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.11.102.61" port: 45917 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 06:11:18 dist-test-slave-9hm5 krb5kdc[1445](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935877, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.102.60@KRBTEST.COM for kudu/127.11.102.61@KRBTEST.COM
Jun 20 06:11:18 dist-test-slave-9hm5 krb5kdc[1445](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935877, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.102.62@KRBTEST.COM
I20260620 06:11:18.025455 1681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:11:18.016446 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.102.61:45917 (local address 127.0.0.1:60358)
0620 06:11:18.016674 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0620 06:11:18.016689 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 06:11:18.016762 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:11:18.017038 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:11:18.017042 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:11:18.017214 (+ 172us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:11:18.017340 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:11:18.017346 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:11:18.018382 (+ 1036us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:11:18.018385 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:11:18.019202 (+ 817us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:11:18.019211 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:11:18.019804 (+ 593us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:11:18.019807 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:11:18.019885 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:11:18.020412 (+ 527us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:11:18.020426 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:11:18.021982 (+ 1556us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:11:18.023946 (+ 1964us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:11:18.023951 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:11:18.023953 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:11:18.024143 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:11:18.024444 (+ 301us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:11:18.024447 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:11:18.024449 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:11:18.024529 (+ 80us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:11:18.024955 (+ 426us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:11:18.024958 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:11:18.025038 (+ 80us) client_negotiation.cc:769] Sending connection context
0620 06:11:18.025122 (+ 84us) client_negotiation.cc:240] Negotiation successful
0620 06:11:18.025265 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":156}
I20260620 06:11:18.025483 1608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:11:18.016543 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60358 (local address 127.11.102.61:45917)
0620 06:11:18.016698 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0620 06:11:18.016703 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 06:11:18.016736 (+ 33us) server_negotiation.cc:407] Connection header received
0620 06:11:18.016860 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:11:18.016864 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:11:18.016923 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:11:18.017003 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:11:18.017445 (+ 442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:11:18.018281 (+ 836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:11:18.019317 (+ 1036us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:11:18.019702 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:11:18.019806 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:11:18.022121 (+ 2315us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:11:18.022125 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:11:18.022128 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:11:18.022157 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:11:18.023839 (+ 1682us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:11:18.024263 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:11:18.024266 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:11:18.024268 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:11:18.024348 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:11:18.024629 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:11:18.024632 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:11:18.024634 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:11:18.024822 (+ 188us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:11:18.024939 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0620 06:11:18.025211 (+ 272us) server_negotiation.cc:299] Negotiation successful
0620 06:11:18.025329 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260620 06:11:18.026502 1530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:11:18.017190 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46330 (local address 127.11.102.62:41463)
0620 06:11:18.017308 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0620 06:11:18.017314 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 06:11:18.017329 (+ 15us) server_negotiation.cc:407] Connection header received
0620 06:11:18.017465 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:11:18.017468 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:11:18.017517 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:11:18.017575 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:11:18.018272 (+ 697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:11:18.019026 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:11:18.020411 (+ 1385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:11:18.020742 (+ 331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:11:18.020857 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:11:18.023355 (+ 2498us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:11:18.023357 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:11:18.023359 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:11:18.023380 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:11:18.024805 (+ 1425us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:11:18.025241 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:11:18.025245 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:11:18.025247 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:11:18.025348 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:11:18.025662 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:11:18.025666 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:11:18.025667 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:11:18.025852 (+ 185us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:11:18.026110 (+ 258us) server_negotiation.cc:1035] Waiting for connection context
0620 06:11:18.026223 (+ 113us) server_negotiation.cc:299] Negotiation successful
0620 06:11:18.026343 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260620 06:11:18.026476 1677 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.11.102.60" port: 44551 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 06:11:18 dist-test-slave-9hm5 krb5kdc[1445](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935877, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.102.60@KRBTEST.COM for kudu/127.11.102.60@KRBTEST.COM
Jun 20 06:11:18 dist-test-slave-9hm5 krb5kdc[1445](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935877, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.102.61@KRBTEST.COM
I20260620 06:11:18.035297 1601 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.11.102.60" port: 44551 } attempt: 3
I20260620 06:11:18.036278 1681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:11:18.027033 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.102.60:44551 (local address 127.0.0.1:58972)
0620 06:11:18.027130 (+ 97us) negotiation.cc:107] Waiting for socket to connect
0620 06:11:18.027142 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 06:11:18.027241 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:11:18.027670 (+ 429us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:11:18.027676 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:11:18.027856 (+ 180us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:11:18.027959 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:11:18.027966 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:11:18.029535 (+ 1569us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:11:18.029537 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:11:18.030032 (+ 495us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:11:18.030039 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:11:18.030589 (+ 550us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:11:18.030593 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:11:18.030668 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:11:18.031228 (+ 560us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:11:18.031241 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:11:18.032535 (+ 1294us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:11:18.034600 (+ 2065us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:11:18.034605 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:11:18.034610 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:11:18.034868 (+ 258us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:11:18.035145 (+ 277us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:11:18.035150 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:11:18.035152 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:11:18.035242 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:11:18.035764 (+ 522us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:11:18.035766 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:11:18.035853 (+ 87us) client_negotiation.cc:769] Sending connection context
0620 06:11:18.035953 (+ 100us) client_negotiation.cc:240] Negotiation successful
0620 06:11:18.036093 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32}
I20260620 06:11:18.036473 1526 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.11.102.60" port: 44551 } attempt: 3
I20260620 06:11:18.036969 1680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:11:18.027336 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58972 (local address 127.11.102.60:44551)
0620 06:11:18.027435 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0620 06:11:18.027440 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 06:11:18.027453 (+ 13us) server_negotiation.cc:407] Connection header received
0620 06:11:18.027487 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:11:18.027489 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:11:18.027534 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:11:18.027646 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:11:18.028086 (+ 440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:11:18.028876 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:11:18.030147 (+ 1271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:11:18.030445 (+ 298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:11:18.030568 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:11:18.032673 (+ 2105us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:11:18.032677 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:11:18.032680 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:11:18.032718 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:11:18.034488 (+ 1770us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:11:18.034967 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:11:18.034977 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:11:18.034979 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:11:18.035046 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:11:18.035384 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:11:18.035387 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:11:18.035388 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:11:18.035607 (+ 219us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:11:18.035729 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0620 06:11:18.036715 (+ 986us) server_negotiation.cc:299] Negotiation successful
0620 06:11:18.036829 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260620 06:11:18.040664 1677 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 76492d24230c4ec5a5d4d934e1bd7930: Bootstrap starting.
Jun 20 06:11:18 dist-test-slave-9hm5 krb5kdc[1445](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935877, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.102.62@KRBTEST.COM for kudu/127.11.102.60@KRBTEST.COM
Jun 20 06:11:18 dist-test-slave-9hm5 krb5kdc[1445](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935877, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.102.61@KRBTEST.COM for kudu/127.11.102.60@KRBTEST.COM
I20260620 06:11:18.043764 1677 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 76492d24230c4ec5a5d4d934e1bd7930: Neither blocks nor log segments found. Creating new log.
I20260620 06:11:18.044579 1677 log.cc:824] T 00000000000000000000000000000000 P 76492d24230c4ec5a5d4d934e1bd7930: Log is configured to *not* fsync() on all Append() calls
I20260620 06:11:18.045095 1608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:11:18.028043 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60360 (local address 127.11.102.61:45917)
0620 06:11:18.028141 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0620 06:11:18.028145 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 06:11:18.028156 (+ 11us) server_negotiation.cc:407] Connection header received
0620 06:11:18.028190 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:11:18.028192 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:11:18.028222 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:11:18.028285 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:11:18.028983 (+ 698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:11:18.029430 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:11:18.030215 (+ 785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:11:18.030417 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:11:18.030493 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:11:18.033343 (+ 2850us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:11:18.033353 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:11:18.033356 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:11:18.033377 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:11:18.034932 (+ 1555us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:11:18.035387 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:11:18.035390 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:11:18.035392 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:11:18.035477 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:11:18.035819 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:11:18.035825 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:11:18.035830 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:11:18.036057 (+ 227us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:11:18.036304 (+ 247us) server_negotiation.cc:1035] Waiting for connection context
0620 06:11:18.036417 (+ 113us) server_negotiation.cc:299] Negotiation successful
0620 06:11:18.036536 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260620 06:11:18.046492 1529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:11:18.036885 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.102.60:44551 (local address 127.0.0.1:58980)
0620 06:11:18.037069 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0620 06:11:18.037082 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 06:11:18.037151 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:11:18.037418 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:11:18.037421 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:11:18.037580 (+ 159us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:11:18.037659 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:11:18.037664 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:11:18.038459 (+ 795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:11:18.038463 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:11:18.039191 (+ 728us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:11:18.039201 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:11:18.039779 (+ 578us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:11:18.039783 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:11:18.039869 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:11:18.040369 (+ 500us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:11:18.040384 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:11:18.042193 (+ 1809us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:11:18.044492 (+ 2299us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:11:18.044496 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:11:18.044498 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:11:18.044750 (+ 252us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:11:18.045042 (+ 292us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:11:18.045046 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:11:18.045048 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:11:18.045139 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:11:18.045942 (+ 803us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:11:18.045945 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:11:18.046045 (+ 100us) client_negotiation.cc:769] Sending connection context
0620 06:11:18.046155 (+ 110us) client_negotiation.cc:240] Negotiation successful
0620 06:11:18.046317 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":107}
I20260620 06:11:18.046599 1604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:11:18.035767 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.102.60:44551 (local address 127.0.0.1:58978)
0620 06:11:18.036222 (+ 455us) negotiation.cc:107] Waiting for socket to connect
0620 06:11:18.036237 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 06:11:18.036313 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:11:18.037072 (+ 759us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:11:18.037078 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:11:18.037266 (+ 188us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:11:18.037369 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:11:18.037376 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:11:18.038780 (+ 1404us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:11:18.038784 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:11:18.039505 (+ 721us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:11:18.039513 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:11:18.040376 (+ 863us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:11:18.040380 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:11:18.040464 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:11:18.040948 (+ 484us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:11:18.040962 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:11:18.042301 (+ 1339us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:11:18.044899 (+ 2598us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:11:18.044901 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:11:18.044904 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:11:18.045153 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:11:18.045480 (+ 327us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:11:18.045483 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:11:18.045485 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:11:18.045576 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:11:18.046098 (+ 522us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:11:18.046101 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:11:18.046203 (+ 102us) client_negotiation.cc:769] Sending connection context
0620 06:11:18.046317 (+ 114us) client_negotiation.cc:240] Negotiation successful
0620 06:11:18.046461 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":377}
I20260620 06:11:18.047142 1682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:11:18.035951 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58978 (local address 127.11.102.60:44551)
0620 06:11:18.036839 (+ 888us) server_negotiation.cc:206] Beginning negotiation
0620 06:11:18.036849 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 06:11:18.036861 (+ 12us) server_negotiation.cc:407] Connection header received
0620 06:11:18.036894 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:11:18.036897 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:11:18.036941 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:11:18.037473 (+ 532us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:11:18.037553 (+ 80us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:11:18.038664 (+ 1111us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:11:18.039659 (+ 995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:11:18.040000 (+ 341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:11:18.040110 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:11:18.042410 (+ 2300us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:11:18.042414 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:11:18.042417 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:11:18.042434 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:11:18.044385 (+ 1951us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:11:18.045270 (+ 885us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:11:18.045273 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:11:18.045275 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:11:18.045376 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:11:18.045689 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:11:18.045695 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:11:18.045699 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:11:18.045990 (+ 291us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:11:18.046088 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0620 06:11:18.046786 (+ 698us) server_negotiation.cc:299] Negotiation successful
0620 06:11:18.046922 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":820,"thread_start_us":112,"threads_started":1}
I20260620 06:11:18.048133 1677 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 76492d24230c4ec5a5d4d934e1bd7930: No bootstrap required, opened a new log
I20260620 06:11:18.048749 11672 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 06:11:18.050165 1601 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e15f6538c09e4081a3a9f37124026a96: Bootstrap starting.
I20260620 06:11:18.049963 1500 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "6ec5bfd83ab54c06920f04ce454a50b6"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46330
I20260620 06:11:18.052057 1680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:11:18.036909 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58980 (local address 127.11.102.60:44551)
0620 06:11:18.037216 (+ 307us) server_negotiation.cc:206] Beginning negotiation
0620 06:11:18.037219 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 06:11:18.037232 (+ 13us) server_negotiation.cc:407] Connection header received
0620 06:11:18.037264 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:11:18.037271 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:11:18.037303 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:11:18.037395 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:11:18.037773 (+ 378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:11:18.038345 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:11:18.039320 (+ 975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:11:18.039663 (+ 343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:11:18.039775 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:11:18.042332 (+ 2557us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:11:18.042336 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:11:18.042338 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:11:18.042364 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:11:18.044370 (+ 2006us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:11:18.044866 (+ 496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:11:18.044868 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:11:18.044870 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:11:18.044940 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:11:18.045289 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:11:18.045292 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:11:18.045294 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:11:18.045829 (+ 535us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:11:18.045931 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0620 06:11:18.051759 (+ 5828us) server_negotiation.cc:299] Negotiation successful
0620 06:11:18.051888 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":242,"thread_start_us":118,"threads_started":1}
I20260620 06:11:18.052321 1677 raft_consensus.cc:348] T 00000000000000000000000000000000 P 76492d24230c4ec5a5d4d934e1bd7930 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ec5bfd83ab54c06920f04ce454a50b6" member_type: VOTER last_known_addr { host: "127.11.102.62" port: 41463 } } peers { permanent_uuid: "e15f6538c09e4081a3a9f37124026a96" member_type: VOTER last_known_addr { host: "127.11.102.61" port: 45917 } } peers { permanent_uuid: "76492d24230c4ec5a5d4d934e1bd7930" member_type: VOTER last_known_addr { host: "127.11.102.60" port: 44551 } }
I20260620 06:11:18.052565 1677 raft_consensus.cc:729] T 00000000000000000000000000000000 P 76492d24230c4ec5a5d4d934e1bd7930 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76492d24230c4ec5a5d4d934e1bd7930, State: Initialized, Role: FOLLOWER
I20260620 06:11:18.052908 1601 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e15f6538c09e4081a3a9f37124026a96: Neither blocks nor log segments found. Creating new log.
I20260620 06:11:18.052902 1677 consensus_queue.cc:260] T 00000000000000000000000000000000 P 76492d24230c4ec5a5d4d934e1bd7930 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ec5bfd83ab54c06920f04ce454a50b6" member_type: VOTER last_known_addr { host: "127.11.102.62" port: 41463 } } peers { permanent_uuid: "e15f6538c09e4081a3a9f37124026a96" member_type: VOTER last_known_addr { host: "127.11.102.61" port: 45917 } } peers { permanent_uuid: "76492d24230c4ec5a5d4d934e1bd7930" member_type: VOTER last_known_addr { host: "127.11.102.60" port: 44551 } }
I20260620 06:11:18.053494 1685 sys_catalog.cc:455] T 00000000000000000000000000000000 P 76492d24230c4ec5a5d4d934e1bd7930 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ec5bfd83ab54c06920f04ce454a50b6" member_type: VOTER last_known_addr { host: "127.11.102.62" port: 41463 } } peers { permanent_uuid: "e15f6538c09e4081a3a9f37124026a96" member_type: VOTER last_known_addr { host: "127.11.102.61" port: 45917 } } peers { permanent_uuid: "76492d24230c4ec5a5d4d934e1bd7930" member_type: VOTER last_known_addr { host: "127.11.102.60" port: 44551 } } }
I20260620 06:11:18.053737 1685 sys_catalog.cc:458] T 00000000000000000000000000000000 P 76492d24230c4ec5a5d4d934e1bd7930 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 06:11:18.053781 1601 log.cc:824] T 00000000000000000000000000000000 P e15f6538c09e4081a3a9f37124026a96: Log is configured to *not* fsync() on all Append() calls
I20260620 06:11:18.054304 1677 sys_catalog.cc:564] T 00000000000000000000000000000000 P 76492d24230c4ec5a5d4d934e1bd7930 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 06:11:18.055140 1526 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6ec5bfd83ab54c06920f04ce454a50b6: Bootstrap starting.
I20260620 06:11:18.056082 1601 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e15f6538c09e4081a3a9f37124026a96: No bootstrap required, opened a new log
I20260620 06:11:18.058079 1526 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6ec5bfd83ab54c06920f04ce454a50b6: Neither blocks nor log segments found. Creating new log.
I20260620 06:11:18.058969 1526 log.cc:824] T 00000000000000000000000000000000 P 6ec5bfd83ab54c06920f04ce454a50b6: Log is configured to *not* fsync() on all Append() calls
W20260620 06:11:18.060154 1697 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 76492d24230c4ec5a5d4d934e1bd7930: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 06:11:18.060235 1697 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 06:11:18.060434 1601 raft_consensus.cc:348] T 00000000000000000000000000000000 P e15f6538c09e4081a3a9f37124026a96 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ec5bfd83ab54c06920f04ce454a50b6" member_type: VOTER last_known_addr { host: "127.11.102.62" port: 41463 } } peers { permanent_uuid: "e15f6538c09e4081a3a9f37124026a96" member_type: VOTER last_known_addr { host: "127.11.102.61" port: 45917 } } peers { permanent_uuid: "76492d24230c4ec5a5d4d934e1bd7930" member_type: VOTER last_known_addr { host: "127.11.102.60" port: 44551 } }
I20260620 06:11:18.060878 1601 raft_consensus.cc:729] T 00000000000000000000000000000000 P e15f6538c09e4081a3a9f37124026a96 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e15f6538c09e4081a3a9f37124026a96, State: Initialized, Role: FOLLOWER
I20260620 06:11:18.061203 1601 consensus_queue.cc:260] T 00000000000000000000000000000000 P e15f6538c09e4081a3a9f37124026a96 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ec5bfd83ab54c06920f04ce454a50b6" member_type: VOTER last_known_addr { host: "127.11.102.62" port: 41463 } } peers { permanent_uuid: "e15f6538c09e4081a3a9f37124026a96" member_type: VOTER last_known_addr { host: "127.11.102.61" port: 45917 } } peers { permanent_uuid: "76492d24230c4ec5a5d4d934e1bd7930" member_type: VOTER last_known_addr { host: "127.11.102.60" port: 44551 } }
I20260620 06:11:18.062469 1601 sys_catalog.cc:564] T 00000000000000000000000000000000 P e15f6538c09e4081a3a9f37124026a96 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 06:11:18.062703 1526 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6ec5bfd83ab54c06920f04ce454a50b6: No bootstrap required, opened a new log
I20260620 06:11:18.060766 1607 master_runner.cc:186] Error getting master registration for 127.11.102.62:41463: OK, instance_id { permanent_uuid: "6ec5bfd83ab54c06920f04ce454a50b6" instance_seqno: 1781935877571636 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260620 06:11:18.064236 1607 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260620 06:11:18.065371 1700 sys_catalog.cc:455] T 00000000000000000000000000000000 P e15f6538c09e4081a3a9f37124026a96 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ec5bfd83ab54c06920f04ce454a50b6" member_type: VOTER last_known_addr { host: "127.11.102.62" port: 41463 } } peers { permanent_uuid: "e15f6538c09e4081a3a9f37124026a96" member_type: VOTER last_known_addr { host: "127.11.102.61" port: 45917 } } peers { permanent_uuid: "76492d24230c4ec5a5d4d934e1bd7930" member_type: VOTER last_known_addr { host: "127.11.102.60" port: 44551 } } }
I20260620 06:11:18.065487 1700 sys_catalog.cc:458] T 00000000000000000000000000000000 P e15f6538c09e4081a3a9f37124026a96 [sys.catalog]: This master's current role is: FOLLOWER
W20260620 06:11:18.068284 1711 catalog_manager.cc:1558] T 00000000000000000000000000000000 P e15f6538c09e4081a3a9f37124026a96: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 06:11:18.068362 1711 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 06:11:18.066310 1533 master_runner.cc:186] Error getting master registration for 127.11.102.62:41463: OK, instance_id { permanent_uuid: "6ec5bfd83ab54c06920f04ce454a50b6" instance_seqno: 1781935877571636 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260620 06:11:18.069702 1533 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260620 06:11:18.071319 1526 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6ec5bfd83ab54c06920f04ce454a50b6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ec5bfd83ab54c06920f04ce454a50b6" member_type: VOTER last_known_addr { host: "127.11.102.62" port: 41463 } } peers { permanent_uuid: "e15f6538c09e4081a3a9f37124026a96" member_type: VOTER last_known_addr { host: "127.11.102.61" port: 45917 } } peers { permanent_uuid: "76492d24230c4ec5a5d4d934e1bd7930" member_type: VOTER last_known_addr { host: "127.11.102.60" port: 44551 } }
I20260620 06:11:18.071558 1526 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6ec5bfd83ab54c06920f04ce454a50b6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6ec5bfd83ab54c06920f04ce454a50b6, State: Initialized, Role: FOLLOWER
I20260620 06:11:18.071931 1526 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6ec5bfd83ab54c06920f04ce454a50b6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ec5bfd83ab54c06920f04ce454a50b6" member_type: VOTER last_known_addr { host: "127.11.102.62" port: 41463 } } peers { permanent_uuid: "e15f6538c09e4081a3a9f37124026a96" member_type: VOTER last_known_addr { host: "127.11.102.61" port: 45917 } } peers { permanent_uuid: "76492d24230c4ec5a5d4d934e1bd7930" member_type: VOTER last_known_addr { host: "127.11.102.60" port: 44551 } }
I20260620 06:11:18.072525 1713 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6ec5bfd83ab54c06920f04ce454a50b6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ec5bfd83ab54c06920f04ce454a50b6" member_type: VOTER last_known_addr { host: "127.11.102.62" port: 41463 } } peers { permanent_uuid: "e15f6538c09e4081a3a9f37124026a96" member_type: VOTER last_known_addr { host: "127.11.102.61" port: 45917 } } peers { permanent_uuid: "76492d24230c4ec5a5d4d934e1bd7930" member_type: VOTER last_known_addr { host: "127.11.102.60" port: 44551 } } }
I20260620 06:11:18.072751 1713 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6ec5bfd83ab54c06920f04ce454a50b6 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 06:11:18.073344 1526 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6ec5bfd83ab54c06920f04ce454a50b6 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 06:11:18.073402 1500 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6ec5bfd83ab54c06920f04ce454a50b6 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260620 06:11:18.073522 1500 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6ec5bfd83ab54c06920f04ce454a50b6 [term 0 FOLLOWER]: Advancing to term 1
I20260620 06:11:18.074817 1500 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6ec5bfd83ab54c06920f04ce454a50b6 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ec5bfd83ab54c06920f04ce454a50b6" member_type: VOTER last_known_addr { host: "127.11.102.62" port: 41463 } } peers { permanent_uuid: "e15f6538c09e4081a3a9f37124026a96" member_type: VOTER last_known_addr { host: "127.11.102.61" port: 45917 } } peers { permanent_uuid: "76492d24230c4ec5a5d4d934e1bd7930" member_type: VOTER last_known_addr { host: "127.11.102.60" port: 44551 } }
I20260620 06:11:18.075863 1500 leader_election.cc:288] T 00000000000000000000000000000000 P 6ec5bfd83ab54c06920f04ce454a50b6 [CANDIDATE]: Term 1 election: Requested vote from peers e15f6538c09e4081a3a9f37124026a96 (127.11.102.61:45917), 76492d24230c4ec5a5d4d934e1bd7930 (127.11.102.60:44551)
I20260620 06:11:18.076630 1575 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6ec5bfd83ab54c06920f04ce454a50b6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "e15f6538c09e4081a3a9f37124026a96"
I20260620 06:11:18.076869 1575 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e15f6538c09e4081a3a9f37124026a96 [term 0 FOLLOWER]: Advancing to term 1
I20260620 06:11:18.078184 1575 raft_consensus.cc:2454] T 00000000000000000000000000000000 P e15f6538c09e4081a3a9f37124026a96 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6ec5bfd83ab54c06920f04ce454a50b6 in term 1.
I20260620 06:11:18.078689 1474 leader_election.cc:302] T 00000000000000000000000000000000 P 6ec5bfd83ab54c06920f04ce454a50b6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6ec5bfd83ab54c06920f04ce454a50b6, e15f6538c09e4081a3a9f37124026a96; no voters:
I20260620 06:11:18.078980 1713 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6ec5bfd83ab54c06920f04ce454a50b6 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 06:11:18.079300 1651 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6ec5bfd83ab54c06920f04ce454a50b6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "76492d24230c4ec5a5d4d934e1bd7930"
I20260620 06:11:18.079555 1651 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 76492d24230c4ec5a5d4d934e1bd7930 [term 0 FOLLOWER]: Advancing to term 1
I20260620 06:11:18.079592 1713 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6ec5bfd83ab54c06920f04ce454a50b6 [term 1 LEADER]: Becoming Leader. State: Replica: 6ec5bfd83ab54c06920f04ce454a50b6, State: Running, Role: LEADER
I20260620 06:11:18.080003 1713 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6ec5bfd83ab54c06920f04ce454a50b6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ec5bfd83ab54c06920f04ce454a50b6" member_type: VOTER last_known_addr { host: "127.11.102.62" port: 41463 } } peers { permanent_uuid: "e15f6538c09e4081a3a9f37124026a96" member_type: VOTER last_known_addr { host: "127.11.102.61" port: 45917 } } peers { permanent_uuid: "76492d24230c4ec5a5d4d934e1bd7930" member_type: VOTER last_known_addr { host: "127.11.102.60" port: 44551 } }
I20260620 06:11:18.080891 1651 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 76492d24230c4ec5a5d4d934e1bd7930 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6ec5bfd83ab54c06920f04ce454a50b6 in term 1.
I20260620 06:11:18.081812 1713 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6ec5bfd83ab54c06920f04ce454a50b6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6ec5bfd83ab54c06920f04ce454a50b6. Latest consensus state: current_term: 1 leader_uuid: "6ec5bfd83ab54c06920f04ce454a50b6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ec5bfd83ab54c06920f04ce454a50b6" member_type: VOTER last_known_addr { host: "127.11.102.62" port: 41463 } } peers { permanent_uuid: "e15f6538c09e4081a3a9f37124026a96" member_type: VOTER last_known_addr { host: "127.11.102.61" port: 45917 } } peers { permanent_uuid: "76492d24230c4ec5a5d4d934e1bd7930" member_type: VOTER last_known_addr { host: "127.11.102.60" port: 44551 } } }
I20260620 06:11:18.081924 1713 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6ec5bfd83ab54c06920f04ce454a50b6 [sys.catalog]: This master's current role is: LEADER
I20260620 06:11:18.088011 1458 master_runner.cc:263] No leader master found from master 6ec5bfd83ab54c06920f04ce454a50b6
I20260620 06:11:18.088097 1458 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260620 06:11:18.088160 1724 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 6ec5bfd83ab54c06920f04ce454a50b6: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 06:11:18.088246 1724 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 06:11:18.097067 1731 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 06:11:18.097929 1680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:11:18.084497 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58984 (local address 127.11.102.60:44551)
0620 06:11:18.084696 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0620 06:11:18.084701 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 06:11:18.088813 (+ 4112us) server_negotiation.cc:407] Connection header received
0620 06:11:18.088976 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:11:18.088979 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:11:18.089021 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:11:18.089069 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:11:18.089924 (+ 855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:11:18.090720 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:11:18.091921 (+ 1201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:11:18.092275 (+ 354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:11:18.092347 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:11:18.094222 (+ 1875us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:11:18.094226 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:11:18.094230 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:11:18.094256 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:11:18.095988 (+ 1732us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:11:18.096512 (+ 524us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:11:18.096516 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:11:18.096518 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:11:18.096600 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:11:18.096898 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:11:18.096902 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:11:18.096904 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:11:18.097115 (+ 211us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:11:18.097222 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0620 06:11:18.097640 (+ 418us) server_negotiation.cc:299] Negotiation successful
0620 06:11:18.097764 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":111}
I20260620 06:11:18.098134 1731 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 06:11:18.103457 1530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:11:18.083990 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46342 (local address 127.11.102.62:41463)
0620 06:11:18.084114 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0620 06:11:18.084119 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 06:11:18.087891 (+ 3772us) server_negotiation.cc:407] Connection header received
0620 06:11:18.088078 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:11:18.088081 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:11:18.088128 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:11:18.088181 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:11:18.095730 (+ 7549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:11:18.096526 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:11:18.097647 (+ 1121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:11:18.097964 (+ 317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:11:18.098043 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:11:18.099841 (+ 1798us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:11:18.099845 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:11:18.099848 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:11:18.099873 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:11:18.101584 (+ 1711us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:11:18.102061 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:11:18.102064 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:11:18.102066 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:11:18.102148 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:11:18.102450 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:11:18.102454 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:11:18.102455 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:11:18.102655 (+ 200us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:11:18.102762 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0620 06:11:18.103119 (+ 357us) server_negotiation.cc:299] Negotiation successful
0620 06:11:18.103296 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260620 06:11:18.107738 1608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:11:18.085080 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60372 (local address 127.11.102.61:45917)
0620 06:11:18.085184 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0620 06:11:18.085188 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 06:11:18.088866 (+ 3678us) server_negotiation.cc:407] Connection header received
0620 06:11:18.089057 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:11:18.089060 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:11:18.089121 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:11:18.089175 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:11:18.095729 (+ 6554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:11:18.096526 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:11:18.101047 (+ 4521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:11:18.101359 (+ 312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:11:18.101442 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:11:18.103741 (+ 2299us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:11:18.103745 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:11:18.103748 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:11:18.103793 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:11:18.105873 (+ 2080us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:11:18.106365 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:11:18.106369 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:11:18.106371 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:11:18.106457 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:11:18.106735 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:11:18.106738 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:11:18.106740 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:11:18.106948 (+ 208us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:11:18.107056 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0620 06:11:18.107400 (+ 344us) server_negotiation.cc:299] Negotiation successful
0620 06:11:18.107525 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260620 06:11:18.121112 1651 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 76492d24230c4ec5a5d4d934e1bd7930 [term 1 FOLLOWER]: Refusing update from remote peer 6ec5bfd83ab54c06920f04ce454a50b6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 06:11:18.121971 1713 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 6ec5bfd83ab54c06920f04ce454a50b6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "76492d24230c4ec5a5d4d934e1bd7930" member_type: VOTER last_known_addr { host: "127.11.102.60" port: 44551 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 06:11:18.123135 1575 raft_consensus.cc:1261] T 00000000000000000000000000000000 P e15f6538c09e4081a3a9f37124026a96 [term 1 FOLLOWER]: Refusing update from remote peer 6ec5bfd83ab54c06920f04ce454a50b6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 06:11:18.124006 1725 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 6ec5bfd83ab54c06920f04ce454a50b6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e15f6538c09e4081a3a9f37124026a96" member_type: VOTER last_known_addr { host: "127.11.102.61" port: 45917 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 06:11:18.131896 1685 sys_catalog.cc:455] T 00000000000000000000000000000000 P 76492d24230c4ec5a5d4d934e1bd7930 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6ec5bfd83ab54c06920f04ce454a50b6. Latest consensus state: current_term: 1 leader_uuid: "6ec5bfd83ab54c06920f04ce454a50b6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ec5bfd83ab54c06920f04ce454a50b6" member_type: VOTER last_known_addr { host: "127.11.102.62" port: 41463 } } peers { permanent_uuid: "e15f6538c09e4081a3a9f37124026a96" member_type: VOTER last_known_addr { host: "127.11.102.61" port: 45917 } } peers { permanent_uuid: "76492d24230c4ec5a5d4d934e1bd7930" member_type: VOTER last_known_addr { host: "127.11.102.60" port: 44551 } } }
I20260620 06:11:18.132022 1685 sys_catalog.cc:458] T 00000000000000000000000000000000 P 76492d24230c4ec5a5d4d934e1bd7930 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 06:11:18.132768 1700 sys_catalog.cc:455] T 00000000000000000000000000000000 P e15f6538c09e4081a3a9f37124026a96 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6ec5bfd83ab54c06920f04ce454a50b6. Latest consensus state: current_term: 1 leader_uuid: "6ec5bfd83ab54c06920f04ce454a50b6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ec5bfd83ab54c06920f04ce454a50b6" member_type: VOTER last_known_addr { host: "127.11.102.62" port: 41463 } } peers { permanent_uuid: "e15f6538c09e4081a3a9f37124026a96" member_type: VOTER last_known_addr { host: "127.11.102.61" port: 45917 } } peers { permanent_uuid: "76492d24230c4ec5a5d4d934e1bd7930" member_type: VOTER last_known_addr { host: "127.11.102.60" port: 44551 } } }
I20260620 06:11:18.132898 1700 sys_catalog.cc:458] T 00000000000000000000000000000000 P e15f6538c09e4081a3a9f37124026a96 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 06:11:18.141852 1713 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6ec5bfd83ab54c06920f04ce454a50b6 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "6ec5bfd83ab54c06920f04ce454a50b6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ec5bfd83ab54c06920f04ce454a50b6" member_type: VOTER last_known_addr { host: "127.11.102.62" port: 41463 } } peers { permanent_uuid: "e15f6538c09e4081a3a9f37124026a96" member_type: VOTER last_known_addr { host: "127.11.102.61" port: 45917 } } peers { permanent_uuid: "76492d24230c4ec5a5d4d934e1bd7930" member_type: VOTER last_known_addr { host: "127.11.102.60" port: 44551 } } }
I20260620 06:11:18.141986 1713 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6ec5bfd83ab54c06920f04ce454a50b6 [sys.catalog]: This master's current role is: LEADER
I20260620 06:11:18.142105 1736 mvcc.cc:205] Tried to move back new op lower bound from 7298809356752044032 to 7298809356619276288. Current Snapshot: MvccSnapshot[applied={T|T < 7298809356752044032}]
I20260620 06:11:18.143138 1685 sys_catalog.cc:455] T 00000000000000000000000000000000 P 76492d24230c4ec5a5d4d934e1bd7930 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "6ec5bfd83ab54c06920f04ce454a50b6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ec5bfd83ab54c06920f04ce454a50b6" member_type: VOTER last_known_addr { host: "127.11.102.62" port: 41463 } } peers { permanent_uuid: "e15f6538c09e4081a3a9f37124026a96" member_type: VOTER last_known_addr { host: "127.11.102.61" port: 45917 } } peers { permanent_uuid: "76492d24230c4ec5a5d4d934e1bd7930" member_type: VOTER last_known_addr { host: "127.11.102.60" port: 44551 } } }
I20260620 06:11:18.143253 1685 sys_catalog.cc:458] T 00000000000000000000000000000000 P 76492d24230c4ec5a5d4d934e1bd7930 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 06:11:18.144363 1725 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6ec5bfd83ab54c06920f04ce454a50b6 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "6ec5bfd83ab54c06920f04ce454a50b6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ec5bfd83ab54c06920f04ce454a50b6" member_type: VOTER last_known_addr { host: "127.11.102.62" port: 41463 } } peers { permanent_uuid: "e15f6538c09e4081a3a9f37124026a96" member_type: VOTER last_known_addr { host: "127.11.102.61" port: 45917 } } peers { permanent_uuid: "76492d24230c4ec5a5d4d934e1bd7930" member_type: VOTER last_known_addr { host: "127.11.102.60" port: 44551 } } }
I20260620 06:11:18.144480 1725 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6ec5bfd83ab54c06920f04ce454a50b6 [sys.catalog]: This master's current role is: LEADER
I20260620 06:11:18.145504 1700 sys_catalog.cc:455] T 00000000000000000000000000000000 P e15f6538c09e4081a3a9f37124026a96 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "6ec5bfd83ab54c06920f04ce454a50b6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ec5bfd83ab54c06920f04ce454a50b6" member_type: VOTER last_known_addr { host: "127.11.102.62" port: 41463 } } peers { permanent_uuid: "e15f6538c09e4081a3a9f37124026a96" member_type: VOTER last_known_addr { host: "127.11.102.61" port: 45917 } } peers { permanent_uuid: "76492d24230c4ec5a5d4d934e1bd7930" member_type: VOTER last_known_addr { host: "127.11.102.60" port: 44551 } } }
I20260620 06:11:18.145619 1700 sys_catalog.cc:458] T 00000000000000000000000000000000 P e15f6538c09e4081a3a9f37124026a96 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 06:11:18.147864 1738 mvcc.cc:205] Tried to move back new op lower bound from 7298809356752044032 to 7298809356619276288. Current Snapshot: MvccSnapshot[applied={T|T < 7298809356752044032}]
I20260620 06:11:18.148036 1737 mvcc.cc:205] Tried to move back new op lower bound from 7298809356752044032 to 7298809356619276288. Current Snapshot: MvccSnapshot[applied={T|T < 7298809356752044032}]
I20260620 06:11:18.149150 1731 catalog_manager.cc:1347] Generated new cluster ID: 14900e49999e40af8dc064f0e9ad0f18
I20260620 06:11:18.149216 1731 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 06:11:18.164897 1731 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 06:11:18.170217 1731 catalog_manager.cc:1504] Loading token signing keys...
I20260620 06:11:18.195456 1731 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6ec5bfd83ab54c06920f04ce454a50b6: Generated new TSK 0
I20260620 06:11:18.196365 1731 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260620 06:11:18.232402 1680 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 06:11:18.228972 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58998 (local address 127.11.102.60:44551)
0620 06:11:18.229125 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0620 06:11:18.229130 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 06:11:18.231198 (+ 2068us) server_negotiation.cc:407] Connection header received
0620 06:11:18.231415 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:11:18.231419 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:11:18.231472 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:11:18.231527 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:11:18.232235 (+ 708us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58998: BlockingRecv error: recv got EOF from 127.0.0.1:58998 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
W20260620 06:11:18.234041 1608 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 06:11:18.229819 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60374 (local address 127.11.102.61:45917)
0620 06:11:18.229939 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0620 06:11:18.229944 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 06:11:18.232931 (+ 2987us) server_negotiation.cc:407] Connection header received
0620 06:11:18.233102 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:11:18.233108 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:11:18.233346 (+ 238us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:11:18.233407 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:11:18.233926 (+ 519us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60374: BlockingRecv error: recv got EOF from 127.0.0.1:60374 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
W20260620 06:11:18.235481 1530 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 06:11:18.230591 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46354 (local address 127.11.102.62:41463)
0620 06:11:18.230704 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0620 06:11:18.230709 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 06:11:18.234520 (+ 3811us) server_negotiation.cc:407] Connection header received
0620 06:11:18.234701 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:11:18.234704 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:11:18.234753 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:11:18.234804 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:11:18.235365 (+ 561us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46354: BlockingRecv error: recv got EOF from 127.0.0.1:46354 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
W20260620 06:11:18.240394 1608 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 06:11:18.237980 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60390 (local address 127.11.102.61:45917)
0620 06:11:18.238846 (+ 866us) server_negotiation.cc:206] Beginning negotiation
0620 06:11:18.238851 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 06:11:18.239319 (+ 468us) server_negotiation.cc:407] Connection header received
0620 06:11:18.239533 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:11:18.239537 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:11:18.239676 (+ 139us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:11:18.239730 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:11:18.240255 (+ 525us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60390: BlockingRecv error: recv got EOF from 127.0.0.1:60390 (error 108)
Metrics: {"server-negotiator.queue_time_us":784}
W20260620 06:11:18.245868 1608 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 06:11:18.244387 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60394 (local address 127.11.102.61:45917)
0620 06:11:18.244489 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0620 06:11:18.244493 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 06:11:18.244962 (+ 469us) server_negotiation.cc:407] Connection header received
0620 06:11:18.245152 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:11:18.245155 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:11:18.245204 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:11:18.245259 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:11:18.245774 (+ 515us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60394: BlockingRecv error: recv got EOF from 127.0.0.1:60394 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
W20260620 06:11:18.251580 1608 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 06:11:18.250067 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60402 (local address 127.11.102.61:45917)
0620 06:11:18.250167 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0620 06:11:18.250171 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 06:11:18.250666 (+ 495us) server_negotiation.cc:407] Connection header received
0620 06:11:18.250854 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:11:18.250857 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:11:18.250907 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:11:18.250960 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:11:18.251478 (+ 518us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60402: BlockingRecv error: recv got EOF from 127.0.0.1:60402 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
W20260620 06:11:18.259548 1608 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 06:11:18.257901 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60404 (local address 127.11.102.61:45917)
0620 06:11:18.258039 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0620 06:11:18.258043 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 06:11:18.258546 (+ 503us) server_negotiation.cc:407] Connection header received
0620 06:11:18.258743 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:11:18.258746 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:11:18.258797 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:11:18.258854 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:11:18.259444 (+ 590us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60404: BlockingRecv error: recv got EOF from 127.0.0.1:60404 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
W20260620 06:11:18.271763 1608 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 06:11:18.270140 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60406 (local address 127.11.102.61:45917)
0620 06:11:18.270271 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0620 06:11:18.270275 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 06:11:18.270768 (+ 493us) server_negotiation.cc:407] Connection header received
0620 06:11:18.270959 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:11:18.270962 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:11:18.271014 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:11:18.271066 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:11:18.271647 (+ 581us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60406: BlockingRecv error: recv got EOF from 127.0.0.1:60406 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
W20260620 06:11:18.291401 1608 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 06:11:18.290190 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60420 (local address 127.11.102.61:45917)
0620 06:11:18.290331 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0620 06:11:18.290334 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 06:11:18.290572 (+ 238us) server_negotiation.cc:407] Connection header received
0620 06:11:18.290664 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:11:18.290667 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:11:18.290733 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:11:18.290826 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:11:18.291268 (+ 442us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60420: BlockingRecv error: recv got EOF from 127.0.0.1:60420 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
W20260620 06:11:18.326970 1608 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 06:11:18.325645 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60424 (local address 127.11.102.61:45917)
0620 06:11:18.325857 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0620 06:11:18.325862 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 06:11:18.326014 (+ 152us) server_negotiation.cc:407] Connection header received
0620 06:11:18.326195 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:11:18.326199 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:11:18.326253 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:11:18.326347 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:11:18.326827 (+ 480us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60424: BlockingRecv error: recv got EOF from 127.0.0.1:60424 (error 108)
Metrics: {"server-negotiator.queue_time_us":100}
W20260620 06:11:18.394488 1608 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 06:11:18.393227 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60438 (local address 127.11.102.61:45917)
0620 06:11:18.393389 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0620 06:11:18.393394 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 06:11:18.393544 (+ 150us) server_negotiation.cc:407] Connection header received
0620 06:11:18.393731 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:11:18.393735 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:11:18.393786 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:11:18.393876 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:11:18.394344 (+ 468us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60438: BlockingRecv error: recv got EOF from 127.0.0.1:60438 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
W20260620 06:11:18.526073 1608 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 06:11:18.524802 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36814 (local address 127.11.102.61:45917)
0620 06:11:18.524988 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0620 06:11:18.524992 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 06:11:18.525057 (+ 65us) server_negotiation.cc:407] Connection header received
0620 06:11:18.525240 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:11:18.525243 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:11:18.525314 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:11:18.525397 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:11:18.525908 (+ 511us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36814: BlockingRecv error: recv got EOF from 127.0.0.1:36814 (error 108)
Metrics: {"server-negotiator.queue_time_us":82}
W20260620 06:11:18.785372 1608 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 06:11:18.784320 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36828 (local address 127.11.102.61:45917)
0620 06:11:18.784460 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0620 06:11:18.784465 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 06:11:18.784588 (+ 123us) server_negotiation.cc:407] Connection header received
0620 06:11:18.784689 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:11:18.784692 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:11:18.784747 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:11:18.784845 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:11:18.785264 (+ 419us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36828: BlockingRecv error: recv got EOF from 127.0.0.1:36828 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
I20260620 06:11:19.061539 1697 catalog_manager.cc:1259] Loaded cluster ID: 14900e49999e40af8dc064f0e9ad0f18
I20260620 06:11:19.061663 1697 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 76492d24230c4ec5a5d4d934e1bd7930: loading cluster ID for follower catalog manager: success
I20260620 06:11:19.063448 1697 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 76492d24230c4ec5a5d4d934e1bd7930: acquiring CA information for follower catalog manager: success
I20260620 06:11:19.064679 1697 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 76492d24230c4ec5a5d4d934e1bd7930: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 06:11:19.069511 1711 catalog_manager.cc:1259] Loaded cluster ID: 14900e49999e40af8dc064f0e9ad0f18
I20260620 06:11:19.069593 1711 catalog_manager.cc:1552] T 00000000000000000000000000000000 P e15f6538c09e4081a3a9f37124026a96: loading cluster ID for follower catalog manager: success
I20260620 06:11:19.071192 1711 catalog_manager.cc:1574] T 00000000000000000000000000000000 P e15f6538c09e4081a3a9f37124026a96: acquiring CA information for follower catalog manager: success
I20260620 06:11:19.072338 1711 catalog_manager.cc:1602] T 00000000000000000000000000000000 P e15f6538c09e4081a3a9f37124026a96: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 06:11:19.303462 1808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:11:19.299605 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36836 (local address 127.11.102.61:45917)
0620 06:11:19.299889 (+ 284us) server_negotiation.cc:206] Beginning negotiation
0620 06:11:19.299892 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 06:11:19.299909 (+ 17us) server_negotiation.cc:407] Connection header received
0620 06:11:19.300058 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:11:19.300061 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:11:19.300184 (+ 123us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:11:19.300296 (+ 112us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 06:11:19.300514 (+ 218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:11:19.301030 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:11:19.301925 (+ 895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:11:19.302160 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:11:19.302254 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:11:19.302542 (+ 288us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 06:11:19.302996 (+ 454us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 06:11:19.303128 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0620 06:11:19.303191 (+ 63us) server_negotiation.cc:299] Negotiation successful
0620 06:11:19.303244 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":193,"thread_start_us":127,"threads_started":1}
I20260620 06:11:19.310418 1814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:11:19.306877 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55712 (local address 127.11.102.60:44551)
0620 06:11:19.307104 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0620 06:11:19.307110 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 06:11:19.307125 (+ 15us) server_negotiation.cc:407] Connection header received
0620 06:11:19.307215 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:11:19.307217 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:11:19.307326 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:11:19.307448 (+ 122us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 06:11:19.307676 (+ 228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:11:19.308180 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:11:19.308995 (+ 815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:11:19.309221 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:11:19.309329 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:11:19.309616 (+ 287us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 06:11:19.310033 (+ 417us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 06:11:19.310170 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0620 06:11:19.310205 (+ 35us) server_negotiation.cc:299] Negotiation successful
0620 06:11:19.310259 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":146,"thread_start_us":53,"threads_started":1}
I20260620 06:11:19.316900 1808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:11:19.313286 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36852 (local address 127.11.102.61:45917)
0620 06:11:19.313586 (+ 300us) server_negotiation.cc:206] Beginning negotiation
0620 06:11:19.313593 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 06:11:19.313609 (+ 16us) server_negotiation.cc:407] Connection header received
0620 06:11:19.313663 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:11:19.313665 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:11:19.313727 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:11:19.313805 (+ 78us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 06:11:19.314009 (+ 204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:11:19.314765 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:11:19.315789 (+ 1024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:11:19.316006 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:11:19.316115 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:11:19.316464 (+ 349us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 06:11:19.316542 (+ 78us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 06:11:19.316609 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0620 06:11:19.316710 (+ 101us) server_negotiation.cc:299] Negotiation successful
0620 06:11:19.316760 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":216}
I20260620 06:11:19.317570 1814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:11:19.313451 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55724 (local address 127.11.102.60:44551)
0620 06:11:19.313730 (+ 279us) server_negotiation.cc:206] Beginning negotiation
0620 06:11:19.313734 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 06:11:19.314132 (+ 398us) server_negotiation.cc:407] Connection header received
0620 06:11:19.314236 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:11:19.314238 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:11:19.314275 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:11:19.314360 (+ 85us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 06:11:19.314570 (+ 210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:11:19.315221 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:11:19.316389 (+ 1168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:11:19.316668 (+ 279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:11:19.316775 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:11:19.317117 (+ 342us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 06:11:19.317188 (+ 71us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 06:11:19.317237 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0620 06:11:19.317392 (+ 155us) server_negotiation.cc:299] Negotiation successful
0620 06:11:19.317438 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":211}
I20260620 06:11:19.318629 1820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:11:19.313263 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38266 (local address 127.11.102.62:41463)
0620 06:11:19.313884 (+ 621us) server_negotiation.cc:206] Beginning negotiation
0620 06:11:19.313890 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 06:11:19.314504 (+ 614us) server_negotiation.cc:407] Connection header received
0620 06:11:19.314605 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:11:19.314608 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:11:19.314749 (+ 141us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:11:19.314858 (+ 109us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 06:11:19.315178 (+ 320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:11:19.316031 (+ 853us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:11:19.317258 (+ 1227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:11:19.317568 (+ 310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:11:19.317665 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:11:19.317984 (+ 319us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 06:11:19.318301 (+ 317us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 06:11:19.318370 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0620 06:11:19.318481 (+ 111us) server_negotiation.cc:299] Negotiation successful
0620 06:11:19.318526 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":540,"thread_start_us":65,"threads_started":1}
I20260620 06:11:19.326390 11672 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9HMp43/build/debug/bin/kudu with pid 1458
I20260620 06:11:19.332646 11672 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9HMp43/build/debug/bin/kudu with pid 1533
I20260620 06:11:19.338333 11672 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9HMp43/build/debug/bin/kudu with pid 1607
2026-06-20T06:11:19Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (3112 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781935603120653-11672-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 06:11:19 dist-test-slave-9hm5 krb5kdc[1825](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 06:11:19 dist-test-slave-9hm5 krb5kdc[1825](info): set up 2 sockets
Jun 20 06:11:19 dist-test-slave-9hm5 krb5kdc[1825](info): commencing operation
krb5kdc: starting...
W20260620 06:11:20.376024 11672 mini_kdc.cc:121] Time spent starting KDC: real 1.016s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781935603120653-11672-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781935603120653-11672-0/krb5kdc/test-user.keytab.
Jun 20 06:11:20 dist-test-slave-9hm5 krb5kdc[1825](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935880, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T06:11:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T06:11:20Z Disabled control of system clock
WARNING: no policy specified for kudu/127.11.102.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.102.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.102.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781935603120653-11672-0/krb5kdc/kudu_127.11.102.62.keytab.
Entry for principal kudu/127.11.102.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781935603120653-11672-0/krb5kdc/kudu_127.11.102.62.keytab.
I20260620 06:11:20.498167 11672 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9HMp43/build/debug/bin/kudu
/tmp/dist-test-task9HMp43/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781935603120653-11672-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781935603120653-11672-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781935603120653-11672-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781935603120653-11672-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.102.62:39415
--webserver_interface=127.11.102.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781935603120653-11672-0/krb5kdc/kudu_127.11.102.62.keytab
--principal=kudu/127.11.102.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.102.20:37789
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.102.62:39415,127.11.102.61:39357,127.11.102.60:42801
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781935603120653-11672-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781935603120653-11672-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781935603120653-11672-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 06:11:20.611829 1838 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260620 06:11:20.612123 1838 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:11:20.612147 1838 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:11:20.612226 1838 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:11:20.616436 1838 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 06:11:20.616496 1838 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:11:20.616516 1838 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:11:20.616533 1838 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 06:11:20.616549 1838 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 06:11:20.621917 1838 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.102.20:37789
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781935603120653-11672-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781935603120653-11672-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.102.62:39415,127.11.102.61:39357,127.11.102.60:42801
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781935603120653-11672-0/krb5kdc/kudu_127.11.102.62.keytab
--principal=kudu/127.11.102.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.102.62:39415
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781935603120653-11672-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.102.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1838
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781935603120653-11672-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 05:28:48 UTC on ccycloud.quasar-fixhms.root.comops.site
build id 4457020
I20260620 06:11:20.623020 1838 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:11:20.624225 1838 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 06:11:20.632593 1846 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:11:20.632609 1843 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:11:20.632632 1844 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:11:20.633064 1838 server_base.cc:1034] running on GCE node
I20260620 06:11:20.633613 1838 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:11:20.634634 1838 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:11:20.635849 1838 hybrid_clock.cc:648] HybridClock initialized: now 1781935880635832 us; error 36 us; skew 500 ppm
Jun 20 06:11:20 dist-test-slave-9hm5 krb5kdc[1825](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935880, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.102.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 06:11:20.638563 1838 init.cc:377] Logged in from keytab as kudu/127.11.102.62@KRBTEST.COM (short username kudu)
I20260620 06:11:20.639655 1838 webserver.cc:460] Webserver started at http://127.11.102.62:38289/ using document root <none> and password file <none>
I20260620 06:11:20.640245 1838 fs_manager.cc:359] Metadata directory not provided
I20260620 06:11:20.640337 1838 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:11:20.640551 1838 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 06:11:20.642556 1838 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781935603120653-11672-0/minicluster-data/master-0/data/instance:
uuid: "dccea233716e45e2aaea40678c2f6bf1"
format_stamp: "Formatted at 2026-06-20 06:11:20 on dist-test-slave-9hm5"
server_key: "8fd05ef9436bde6a07a3b1944507fd8f"
server_key_iv: "2c42fa325b8f41add41abc6ef29ddce6"
server_key_version: "encryptionkey@0"
I20260620 06:11:20.643183 1838 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781935603120653-11672-0/minicluster-data/master-0/wal/instance:
uuid: "dccea233716e45e2aaea40678c2f6bf1"
format_stamp: "Formatted at 2026-06-20 06:11:20 on dist-test-slave-9hm5"
server_key: "8fd05ef9436bde6a07a3b1944507fd8f"
server_key_iv: "2c42fa325b8f41add41abc6ef29ddce6"
server_key_version: "encryptionkey@0"
I20260620 06:11:20.646935 1838 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260620 06:11:20.649372 1853 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:11:20.650594 1838 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260620 06:11:20.650732 1838 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781935603120653-11672-0/minicluster-data/master-0/data,/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781935603120653-11672-0/minicluster-data/master-0/wal
uuid: "dccea233716e45e2aaea40678c2f6bf1"
format_stamp: "Formatted at 2026-06-20 06:11:20 on dist-test-slave-9hm5"
server_key: "8fd05ef9436bde6a07a3b1944507fd8f"
server_key_iv: "2c42fa325b8f41add41abc6ef29ddce6"
server_key_version: "encryptionkey@0"
I20260620 06:11:20.650871 1838 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781935603120653-11672-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781935603120653-11672-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781935603120653-11672-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:11:20.669952 1838 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:11:20.673049 1838 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:11:20.673281 1838 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:11:20.680486 1838 rpc_server.cc:307] RPC server started. Bound to: 127.11.102.62:39415
I20260620 06:11:20.680764 1905 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.102.62:39415 every 8 connection(s)
I20260620 06:11:20.681557 1838 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781935603120653-11672-0/minicluster-data/master-0/data/info.pb
I20260620 06:11:20.684303 1906 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 06:11:20.684968 11672 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9HMp43/build/debug/bin/kudu as pid 1838
I20260620 06:11:20.685072 11672 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781935603120653-11672-0/minicluster-data/master-0/wal/instance
I20260620 06:11:20.685420 11672 external_mini_cluster.cc:1428] Setting key a5fa74d36941f4402d899bbe6f2dd7a5
I20260620 06:11:20.688124 1906 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.11.102.62" port: 39415 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.11.102.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.102.61@KRBTEST.COM" created.
Jun 20 06:11:20 dist-test-slave-9hm5 krb5kdc[1825](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935880, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.102.62@KRBTEST.COM for kudu/127.11.102.62@KRBTEST.COM
I20260620 06:11:20.704568 1909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:11:20.690326 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.102.62:39415 (local address 127.0.0.1:40260)
0620 06:11:20.690813 (+ 487us) negotiation.cc:107] Waiting for socket to connect
0620 06:11:20.690866 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0620 06:11:20.691844 (+ 978us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:11:20.692691 (+ 847us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:11:20.692704 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:11:20.693096 (+ 392us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:11:20.693355 (+ 259us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:11:20.693368 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:11:20.694719 (+ 1351us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:11:20.694723 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:11:20.695596 (+ 873us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:11:20.695653 (+ 57us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:11:20.696536 (+ 883us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:11:20.696544 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:11:20.696697 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:11:20.697299 (+ 602us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:11:20.697325 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:11:20.699102 (+ 1777us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:11:20.701563 (+ 2461us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:11:20.701571 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:11:20.701580 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:11:20.701813 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:11:20.702192 (+ 379us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:11:20.702194 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:11:20.702195 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:11:20.702321 (+ 126us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:11:20.703111 (+ 790us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:11:20.703117 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:11:20.703267 (+ 150us) client_negotiation.cc:769] Sending connection context
0620 06:11:20.703425 (+ 158us) client_negotiation.cc:240] Negotiation successful
0620 06:11:20.703769 (+ 344us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":279,"thread_start_us":145,"threads_started":1}
I20260620 06:11:20.704571 1910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:11:20.691194 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40260 (local address 127.11.102.62:39415)
0620 06:11:20.691531 (+ 337us) server_negotiation.cc:206] Beginning negotiation
0620 06:11:20.691545 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 06:11:20.691595 (+ 50us) server_negotiation.cc:407] Connection header received
0620 06:11:20.692177 (+ 582us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:11:20.692184 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:11:20.692468 (+ 284us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:11:20.692674 (+ 206us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:11:20.693511 (+ 837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:11:20.694446 (+ 935us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:11:20.695768 (+ 1322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:11:20.696343 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:11:20.696480 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:11:20.699254 (+ 2774us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:11:20.699264 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:11:20.699280 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:11:20.699308 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:11:20.701435 (+ 2127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:11:20.701928 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:11:20.701936 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:11:20.701939 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:11:20.702092 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:11:20.702416 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:11:20.702421 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:11:20.702423 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:11:20.702969 (+ 546us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:11:20.703120 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0620 06:11:20.703545 (+ 425us) server_negotiation.cc:299] Negotiation successful
0620 06:11:20.703906 (+ 361us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":120,"server-negotiator.queue_time_us":219,"thread_start_us":91,"threads_started":1}
I20260620 06:11:20.707293 1906 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.11.102.61" port: 39357 } has no permanent_uuid. Determining permanent_uuid...
I20260620 06:11:20.708173 1909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:11:20.707808 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.102.61:39357 (local address 127.0.0.1:49740)
0620 06:11:20.708016 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0620 06:11:20.708108 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.102.61:39357: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
W20260620 06:11:20.708454 1855 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.11.102.61:39357: connect: Connection refused (error 111)
I20260620 06:11:20.709349 1909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:11:20.709109 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.102.61:39357 (local address 127.0.0.1:49754)
0620 06:11:20.709241 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0620 06:11:20.709295 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.102.61:39357: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
W20260620 06:11:20.709525 1906 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.11.102.61:39357: Network error: Client connection negotiation failed: client connection to 127.11.102.61:39357: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.102.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781935603120653-11672-0/krb5kdc/kudu_127.11.102.61.keytab.
Entry for principal kudu/127.11.102.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781935603120653-11672-0/krb5kdc/kudu_127.11.102.61.keytab.
I20260620 06:11:20.714977 11672 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9HMp43/build/debug/bin/kudu
/tmp/dist-test-task9HMp43/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781935603120653-11672-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781935603120653-11672-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781935603120653-11672-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781935603120653-11672-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.102.61:39357
--webserver_interface=127.11.102.61
--webserver_port=0
--keytab_file=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781935603120653-11672-0/krb5kdc/kudu_127.11.102.61.keytab
--principal=kudu/127.11.102.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.102.20:37789
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.102.62:39415,127.11.102.61:39357,127.11.102.60:42801
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781935603120653-11672-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781935603120653-11672-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781935603120653-11672-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 06:11:20.758785 1906 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.11.102.61" port: 39357 } attempt: 1
I20260620 06:11:20.759797 1909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:11:20.759436 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.102.61:39357 (local address 127.0.0.1:49768)
0620 06:11:20.759591 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0620 06:11:20.759718 (+ 127us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.102.61:39357: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20260620 06:11:20.760557 1909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:11:20.760340 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.102.61:39357 (local address 127.0.0.1:49772)
0620 06:11:20.760460 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0620 06:11:20.760506 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.102.61:39357: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
W20260620 06:11:20.760715 1906 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.11.102.61:39357: Network error: Client connection negotiation failed: client connection to 127.11.102.61:39357: connect: Connection refused (error 111)
W20260620 06:11:20.826996 1913 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260620 06:11:20.827306 1913 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:11:20.827329 1913 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:11:20.827411 1913 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260620 06:11:20.828936 1906 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.11.102.61" port: 39357 } attempt: 2
I20260620 06:11:20.829744 1909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:11:20.829448 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.102.61:39357 (local address 127.0.0.1:49776)
0620 06:11:20.829597 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0620 06:11:20.829661 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.102.61:39357: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260620 06:11:20.830469 1909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:11:20.830257 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.102.61:39357 (local address 127.0.0.1:49786)
0620 06:11:20.830364 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0620 06:11:20.830419 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.102.61:39357: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260620 06:11:20.830595 1906 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.11.102.61:39357: Network error: Client connection negotiation failed: client connection to 127.11.102.61:39357: connect: Connection refused (error 111)
W20260620 06:11:20.832180 1913 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 06:11:20.832257 1913 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:11:20.832306 1913 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:11:20.832348 1913 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 06:11:20.832401 1913 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 06:11:20.837927 1913 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.102.20:37789
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781935603120653-11672-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781935603120653-11672-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.11.102.62:39415,127.11.102.61:39357,127.11.102.60:42801
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781935603120653-11672-0/krb5kdc/kudu_127.11.102.61.keytab
--principal=kudu/127.11.102.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.102.61:39357
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781935603120653-11672-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.11.102.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1913
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781935603120653-11672-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 05:28:48 UTC on ccycloud.quasar-fixhms.root.comops.site
build id 4457020
I20260620 06:11:20.839030 1913 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:11:20.840198 1913 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 06:11:20.849078 1913 server_base.cc:1034] running on GCE node
W20260620 06:11:20.849036 1921 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:11:20.849036 1919 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:11:20.849031 1918 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:11:20.849916 1913 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:11:20.851240 1913 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:11:20.852432 1913 hybrid_clock.cc:648] HybridClock initialized: now 1781935880852382 us; error 66 us; skew 500 ppm
Jun 20 06:11:20 dist-test-slave-9hm5 krb5kdc[1825](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935880, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.102.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 06:11:20.855491 1913 init.cc:377] Logged in from keytab as kudu/127.11.102.61@KRBTEST.COM (short username kudu)
I20260620 06:11:20.856475 1913 webserver.cc:460] Webserver started at http://127.11.102.61:46067/ using document root <none> and password file <none>
I20260620 06:11:20.857050 1913 fs_manager.cc:359] Metadata directory not provided
I20260620 06:11:20.857141 1913 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:11:20.857391 1913 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 06:11:20.859484 1913 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781935603120653-11672-0/minicluster-data/master-1/data/instance:
uuid: "51c2e75c02754412bb63721962c22346"
format_stamp: "Formatted at 2026-06-20 06:11:20 on dist-test-slave-9hm5"
server_key: "72d40d01c83a2e88db775556d4b7ffa2"
server_key_iv: "360d38922795e9a9d711b6193838ba67"
server_key_version: "encryptionkey@0"
I20260620 06:11:20.860183 1913 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781935603120653-11672-0/minicluster-data/master-1/wal/instance:
uuid: "51c2e75c02754412bb63721962c22346"
format_stamp: "Formatted at 2026-06-20 06:11:20 on dist-test-slave-9hm5"
server_key: "72d40d01c83a2e88db775556d4b7ffa2"
server_key_iv: "360d38922795e9a9d711b6193838ba67"
server_key_version: "encryptionkey@0"
I20260620 06:11:20.863986 1913 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260620 06:11:20.866460 1928 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:11:20.867731 1913 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 06:11:20.867871 1913 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781935603120653-11672-0/minicluster-data/master-1/data,/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781935603120653-11672-0/minicluster-data/master-1/wal
uuid: "51c2e75c02754412bb63721962c22346"
format_stamp: "Formatted at 2026-06-20 06:11:20 on dist-test-slave-9hm5"
server_key: "72d40d01c83a2e88db775556d4b7ffa2"
server_key_iv: "360d38922795e9a9d711b6193838ba67"
server_key_version: "encryptionkey@0"
I20260620 06:11:20.867987 1913 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781935603120653-11672-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781935603120653-11672-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781935603120653-11672-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:11:20.880833 1913 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:11:20.884192 1913 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:11:20.884404 1913 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:11:20.891402 1913 rpc_server.cc:307] RPC server started. Bound to: 127.11.102.61:39357
I20260620 06:11:20.891728 1980 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.102.61:39357 every 8 connection(s)
I20260620 06:11:20.892472 1913 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781935603120653-11672-0/minicluster-data/master-1/data/info.pb
I20260620 06:11:20.895211 1981 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 06:11:20.899127 1981 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.11.102.62" port: 39415 } has no permanent_uuid. Determining permanent_uuid...
I20260620 06:11:20.901803 11672 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9HMp43/build/debug/bin/kudu as pid 1913
I20260620 06:11:20.901944 11672 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781935603120653-11672-0/minicluster-data/master-1/wal/instance
I20260620 06:11:20.902244 11672 external_mini_cluster.cc:1428] Setting key 58fe272be21004a2f15d7f7cfe9dd588
Jun 20 06:11:20 dist-test-slave-9hm5 krb5kdc[1825](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935880, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.102.61@KRBTEST.COM for kudu/127.11.102.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.11.102.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.11.102.60@KRBTEST.COM" created.
I20260620 06:11:20.912874 1910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:11:20.901016 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40272 (local address 127.11.102.62:39415)
0620 06:11:20.901159 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0620 06:11:20.901162 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 06:11:20.902009 (+ 847us) server_negotiation.cc:407] Connection header received
0620 06:11:20.903078 (+ 1069us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:11:20.903085 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:11:20.903154 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:11:20.903288 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:11:20.904525 (+ 1237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:11:20.905056 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:11:20.906201 (+ 1145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:11:20.906443 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:11:20.906561 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:11:20.909241 (+ 2680us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:11:20.909245 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:11:20.909252 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:11:20.909277 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:11:20.910789 (+ 1512us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:11:20.911262 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:11:20.911265 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:11:20.911266 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:11:20.911342 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:11:20.911717 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:11:20.911722 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:11:20.911728 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:11:20.911907 (+ 179us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:11:20.911985 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0620 06:11:20.912589 (+ 604us) server_negotiation.cc:299] Negotiation successful
0620 06:11:20.912737 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260620 06:11:20.913970 1983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:11:20.901377 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.102.62:39415 (local address 127.0.0.1:40272)
0620 06:11:20.901857 (+ 480us) negotiation.cc:107] Waiting for socket to connect
0620 06:11:20.901909 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0620 06:11:20.902857 (+ 948us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:11:20.903455 (+ 598us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:11:20.903462 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:11:20.903890 (+ 428us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:11:20.904363 (+ 473us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:11:20.904376 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:11:20.905184 (+ 808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:11:20.905188 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:11:20.906069 (+ 881us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:11:20.906077 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:11:20.906599 (+ 522us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:11:20.906604 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:11:20.906778 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:11:20.907383 (+ 605us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:11:20.907399 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:11:20.909076 (+ 1677us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:11:20.910897 (+ 1821us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:11:20.910901 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:11:20.910912 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:11:20.911158 (+ 246us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:11:20.911418 (+ 260us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:11:20.911420 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:11:20.911421 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:11:20.911590 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:11:20.911998 (+ 408us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:11:20.912002 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:11:20.912227 (+ 225us) client_negotiation.cc:769] Sending connection context
0620 06:11:20.912438 (+ 211us) client_negotiation.cc:240] Negotiation successful
0620 06:11:20.912750 (+ 312us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":287,"thread_start_us":157,"threads_started":1}
I20260620 06:11:20.915388 1981 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.11.102.61" port: 39357 } has no permanent_uuid. Determining permanent_uuid...
I20260620 06:11:20.921756 1906 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.11.102.61" port: 39357 } attempt: 3
Jun 20 06:11:20 dist-test-slave-9hm5 krb5kdc[1825](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935880, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.102.61@KRBTEST.COM for kudu/127.11.102.61@KRBTEST.COM
I20260620 06:11:20.927443 1983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:11:20.915954 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.102.61:39357 (local address 127.0.0.1:49790)
0620 06:11:20.916065 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0620 06:11:20.916080 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 06:11:20.916152 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:11:20.916909 (+ 757us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:11:20.916911 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:11:20.917039 (+ 128us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:11:20.917122 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:11:20.917126 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:11:20.918675 (+ 1549us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:11:20.918677 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:11:20.919200 (+ 523us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:11:20.919207 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:11:20.920110 (+ 903us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:11:20.920112 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:11:20.920235 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:11:20.920741 (+ 506us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:11:20.920752 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:11:20.923599 (+ 2847us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:11:20.925626 (+ 2027us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:11:20.925630 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:11:20.925632 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:11:20.925905 (+ 273us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:11:20.926221 (+ 316us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:11:20.926223 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:11:20.926225 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:11:20.926305 (+ 80us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:11:20.926888 (+ 583us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:11:20.926891 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:11:20.926978 (+ 87us) client_negotiation.cc:769] Sending connection context
0620 06:11:20.927097 (+ 119us) client_negotiation.cc:240] Negotiation successful
0620 06:11:20.927241 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":45,"mutex_wait_us":28}
I20260620 06:11:20.927850 1986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:11:20.916097 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49790 (local address 127.11.102.61:39357)
0620 06:11:20.916444 (+ 347us) server_negotiation.cc:206] Beginning negotiation
0620 06:11:20.916459 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 06:11:20.916486 (+ 27us) server_negotiation.cc:407] Connection header received
0620 06:11:20.916534 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:11:20.916544 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:11:20.916769 (+ 225us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:11:20.916923 (+ 154us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:11:20.917247 (+ 324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:11:20.918555 (+ 1308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:11:20.919315 (+ 760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:11:20.919999 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:11:20.920150 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:11:20.923813 (+ 3663us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:11:20.923825 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:11:20.923842 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:11:20.923866 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:11:20.925521 (+ 1655us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:11:20.926036 (+ 515us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:11:20.926042 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:11:20.926044 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:11:20.926131 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:11:20.926391 (+ 260us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:11:20.926394 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:11:20.926396 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:11:20.926742 (+ 346us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:11:20.926914 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0620 06:11:20.927197 (+ 283us) server_negotiation.cc:299] Negotiation successful
0620 06:11:20.927330 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":26,"server-negotiator.queue_time_us":222,"thread_start_us":107,"threads_started":1}
Jun 20 06:11:20 dist-test-slave-9hm5 krb5kdc[1825](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935880, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.102.62@KRBTEST.COM for kudu/127.11.102.61@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.11.102.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781935603120653-11672-0/krb5kdc/kudu_127.11.102.60.keytab.
Entry for principal kudu/127.11.102.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781935603120653-11672-0/krb5kdc/kudu_127.11.102.60.keytab.
I20260620 06:11:20.930027 1981 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.11.102.60" port: 42801 } has no permanent_uuid. Determining permanent_uuid...
I20260620 06:11:20.930763 1983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:11:20.930501 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.102.60:42801 (local address 127.0.0.1:45958)
0620 06:11:20.930618 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0620 06:11:20.930702 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.102.60:42801: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260620 06:11:20.930883 11672 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9HMp43/build/debug/bin/kudu
/tmp/dist-test-task9HMp43/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781935603120653-11672-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781935603120653-11672-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781935603120653-11672-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781935603120653-11672-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.102.60:42801
--webserver_interface=127.11.102.60
--webserver_port=0
--keytab_file=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781935603120653-11672-0/krb5kdc/kudu_127.11.102.60.keytab
--principal=kudu/127.11.102.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.11.102.20:37789
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.102.62:39415,127.11.102.61:39357,127.11.102.60:42801
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781935603120653-11672-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781935603120653-11672-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781935603120653-11672-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 06:11:20.931033 1931 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.11.102.60:42801: connect: Connection refused (error 111)
I20260620 06:11:20.932412 1983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:11:20.932178 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.102.60:42801 (local address 127.0.0.1:45970)
0620 06:11:20.932290 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0620 06:11:20.932348 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.102.60:42801: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
I20260620 06:11:20.932655 1909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:11:20.922236 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.102.61:39357 (local address 127.0.0.1:49798)
0620 06:11:20.922373 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0620 06:11:20.922388 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 06:11:20.922468 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:11:20.922804 (+ 336us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:11:20.922808 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:11:20.922971 (+ 163us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:11:20.923068 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:11:20.923073 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:11:20.924404 (+ 1331us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:11:20.924406 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:11:20.925892 (+ 1486us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:11:20.925904 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:11:20.926507 (+ 603us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:11:20.926511 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:11:20.926608 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:11:20.927059 (+ 451us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:11:20.927071 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:11:20.928557 (+ 1486us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:11:20.930971 (+ 2414us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:11:20.930974 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:11:20.930976 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:11:20.931238 (+ 262us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:11:20.931578 (+ 340us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:11:20.931582 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:11:20.931586 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:11:20.931728 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:11:20.932135 (+ 407us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:11:20.932138 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:11:20.932243 (+ 105us) client_negotiation.cc:769] Sending connection context
0620 06:11:20.932348 (+ 105us) client_negotiation.cc:240] Negotiation successful
0620 06:11:20.932496 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":60}
W20260620 06:11:20.932595 1981 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.11.102.60:42801: Network error: Client connection negotiation failed: client connection to 127.11.102.60:42801: connect: Connection refused (error 111)
I20260620 06:11:20.933207 1987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:11:20.922309 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49798 (local address 127.11.102.61:39357)
0620 06:11:20.922557 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0620 06:11:20.922563 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 06:11:20.922573 (+ 10us) server_negotiation.cc:407] Connection header received
0620 06:11:20.922601 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:11:20.922603 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:11:20.922668 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:11:20.922777 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:11:20.923179 (+ 402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:11:20.924025 (+ 846us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:11:20.926022 (+ 1997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:11:20.926390 (+ 368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:11:20.926485 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:11:20.928762 (+ 2277us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:11:20.928766 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:11:20.928768 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:11:20.928789 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:11:20.930866 (+ 2077us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:11:20.931359 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:11:20.931365 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:11:20.931370 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:11:20.931465 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:11:20.931831 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:11:20.931834 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:11:20.931836 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:11:20.932026 (+ 190us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:11:20.932133 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0620 06:11:20.933004 (+ 871us) server_negotiation.cc:299] Negotiation successful
0620 06:11:20.933098 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":187,"thread_start_us":103,"threads_started":1}
I20260620 06:11:20.933842 1906 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.11.102.60" port: 42801 } has no permanent_uuid. Determining permanent_uuid...
I20260620 06:11:20.934577 1909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:11:20.934272 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.102.60:42801 (local address 127.0.0.1:45976)
0620 06:11:20.934420 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0620 06:11:20.934485 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.102.60:42801: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260620 06:11:20.935416 1909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:11:20.935097 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.102.60:42801 (local address 127.0.0.1:45978)
0620 06:11:20.935267 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0620 06:11:20.935330 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.102.60:42801: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81}
W20260620 06:11:20.935559 1906 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.11.102.60:42801: Network error: Client connection negotiation failed: client connection to 127.11.102.60:42801: connect: Connection refused (error 111)
I20260620 06:11:20.966761 1906 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.11.102.60" port: 42801 } attempt: 1
I20260620 06:11:20.967689 1909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:11:20.967324 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.102.60:42801 (local address 127.0.0.1:45990)
0620 06:11:20.967500 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0620 06:11:20.967575 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.102.60:42801: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260620 06:11:20.968461 1909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:11:20.968235 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.102.60:42801 (local address 127.0.0.1:45992)
0620 06:11:20.968345 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0620 06:11:20.968397 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.102.60:42801: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260620 06:11:20.968597 1906 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.11.102.60:42801: Network error: Client connection negotiation failed: client connection to 127.11.102.60:42801: connect: Connection refused (error 111)
I20260620 06:11:20.981900 1981 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.11.102.60" port: 42801 } attempt: 1
I20260620 06:11:20.982743 1983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:11:20.982449 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.102.60:42801 (local address 127.0.0.1:46002)
0620 06:11:20.982588 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0620 06:11:20.982661 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.102.60:42801: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260620 06:11:20.983547 1983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:11:20.983330 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.102.60:42801 (local address 127.0.0.1:46014)
0620 06:11:20.983436 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0620 06:11:20.983491 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.102.60:42801: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260620 06:11:20.983727 1981 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.11.102.60:42801: Network error: Client connection negotiation failed: client connection to 127.11.102.60:42801: connect: Connection refused (error 111)
I20260620 06:11:21.043752 1906 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.11.102.60" port: 42801 } attempt: 2
I20260620 06:11:21.044618 1909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:11:21.044296 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.102.60:42801 (local address 127.0.0.1:46018)
0620 06:11:21.044458 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0620 06:11:21.044528 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.102.60:42801: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260620 06:11:21.045343 1909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:11:21.045127 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.102.60:42801 (local address 127.0.0.1:46032)
0620 06:11:21.045236 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0620 06:11:21.045285 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.102.60:42801: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260620 06:11:21.045553 1906 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.11.102.60:42801: Network error: Client connection negotiation failed: client connection to 127.11.102.60:42801: connect: Connection refused (error 111)
W20260620 06:11:21.046800 1989 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260620 06:11:21.047072 1989 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:11:21.047159 1989 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:11:21.047260 1989 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:11:21.051451 1989 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 06:11:21.051517 1989 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:11:21.051537 1989 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:11:21.051554 1989 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 06:11:21.051627 1989 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 06:11:21.051892 1981 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.11.102.60" port: 42801 } attempt: 2
I20260620 06:11:21.052670 1983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:11:21.052404 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.102.60:42801 (local address 127.0.0.1:46034)
0620 06:11:21.052539 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0620 06:11:21.052598 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.102.60:42801: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260620 06:11:21.053434 1983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:11:21.053218 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.102.60:42801 (local address 127.0.0.1:46040)
0620 06:11:21.053317 (+ 99us) negotiation.cc:107] Waiting for socket to connect
0620 06:11:21.053364 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.11.102.60:42801: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260620 06:11:21.053589 1981 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.11.102.60:42801: Network error: Client connection negotiation failed: client connection to 127.11.102.60:42801: connect: Connection refused (error 111)
I20260620 06:11:21.057688 1989 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.102.20:37789
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781935603120653-11672-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781935603120653-11672-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.11.102.62:39415,127.11.102.61:39357,127.11.102.60:42801
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781935603120653-11672-0/krb5kdc/kudu_127.11.102.60.keytab
--principal=kudu/127.11.102.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.102.60:42801
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781935603120653-11672-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.11.102.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1989
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781935603120653-11672-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 05:28:48 UTC on ccycloud.quasar-fixhms.root.comops.site
build id 4457020
I20260620 06:11:21.058801 1989 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:11:21.059970 1989 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 06:11:21.069666 1994 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:11:21.069867 1995 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:11:21.069944 1997 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:11:21.070326 1989 server_base.cc:1034] running on GCE node
I20260620 06:11:21.071178 1989 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:11:21.072757 1989 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:11:21.076036 1989 hybrid_clock.cc:648] HybridClock initialized: now 1781935881075977 us; error 109 us; skew 500 ppm
Jun 20 06:11:21 dist-test-slave-9hm5 krb5kdc[1825](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935881, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.102.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 06:11:21.079689 1989 init.cc:377] Logged in from keytab as kudu/127.11.102.60@KRBTEST.COM (short username kudu)
I20260620 06:11:21.080792 1989 webserver.cc:460] Webserver started at http://127.11.102.60:39993/ using document root <none> and password file <none>
I20260620 06:11:21.081465 1989 fs_manager.cc:359] Metadata directory not provided
I20260620 06:11:21.081525 1989 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:11:21.081764 1989 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 06:11:21.084868 1989 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781935603120653-11672-0/minicluster-data/master-2/data/instance:
uuid: "501d32ad709a4a42a12a3f712695adbf"
format_stamp: "Formatted at 2026-06-20 06:11:21 on dist-test-slave-9hm5"
server_key: "640e4b8f87c5c06ab1db2bfa36a348bb"
server_key_iv: "d0eceba5a58afa1dfa173338e5282b13"
server_key_version: "encryptionkey@0"
I20260620 06:11:21.085690 1989 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781935603120653-11672-0/minicluster-data/master-2/wal/instance:
uuid: "501d32ad709a4a42a12a3f712695adbf"
format_stamp: "Formatted at 2026-06-20 06:11:21 on dist-test-slave-9hm5"
server_key: "640e4b8f87c5c06ab1db2bfa36a348bb"
server_key_iv: "d0eceba5a58afa1dfa173338e5282b13"
server_key_version: "encryptionkey@0"
I20260620 06:11:21.091027 1989 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260620 06:11:21.094450 2004 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:11:21.096174 1989 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 06:11:21.096324 1989 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781935603120653-11672-0/minicluster-data/master-2/data,/tmp/dist-test-task9HMp43/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781935603120653-11672-0/minicluster-data/master-2/wal
uuid: "501d32ad709a4a42a12a3f712695adbf"
format_stamp: "Formatted at 2026-06-20 06:11:21 on dist-test-slave-9hm5"
server_key: "640e4b8f87c5c06ab1db2bfa36a348bb"
server_key_iv: "d0eceba5a58afa1dfa173338e5282b13"
server_key_version: "encryptionkey@0"
I20260620 06:11:21.096457 1989 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781935603120653-11672-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781935603120653-11672-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781935603120653-11672-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:11:21.124166 1989 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:11:21.127074 1989 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:11:21.127292 1989 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:11:21.134151 1989 rpc_server.cc:307] RPC server started. Bound to: 127.11.102.60:42801
I20260620 06:11:21.134456 2056 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.11.102.60:42801 every 8 connection(s)
I20260620 06:11:21.135126 1989 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781935603120653-11672-0/minicluster-data/master-2/data/info.pb
I20260620 06:11:21.137874 2057 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 06:11:21.138238 11672 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9HMp43/build/debug/bin/kudu as pid 1989
I20260620 06:11:21.138370 11672 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9HMp43/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781935603120653-11672-0/minicluster-data/master-2/wal/instance
I20260620 06:11:21.138617 11672 external_mini_cluster.cc:1428] Setting key 4e2461a5adefea409bf101d01c896291
I20260620 06:11:21.142931 2057 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.11.102.62" port: 39415 } has no permanent_uuid. Determining permanent_uuid...
I20260620 06:11:21.144717 1906 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.11.102.60" port: 42801 } attempt: 3
I20260620 06:11:21.144773 1981 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.11.102.60" port: 42801 } attempt: 3
Jun 20 06:11:21 dist-test-slave-9hm5 krb5kdc[1825](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935880, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.102.60@KRBTEST.COM
Jun 20 06:11:21 dist-test-slave-9hm5 krb5kdc[1825](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935880, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.102.61@KRBTEST.COM for kudu/127.11.102.60@KRBTEST.COM
Jun 20 06:11:21 dist-test-slave-9hm5 krb5kdc[1825](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935880, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.102.62@KRBTEST.COM for kudu/127.11.102.60@KRBTEST.COM
Jun 20 06:11:21 dist-test-slave-9hm5 krb5kdc[1825](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935881, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.102.60@KRBTEST.COM for kudu/127.11.102.62@KRBTEST.COM
I20260620 06:11:21.156632 1983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:11:21.145185 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.102.60:42801 (local address 127.0.0.1:46070)
0620 06:11:21.145411 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0620 06:11:21.145438 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0620 06:11:21.145519 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:11:21.146531 (+ 1012us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:11:21.146534 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:11:21.146737 (+ 203us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:11:21.146858 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:11:21.146868 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:11:21.147903 (+ 1035us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:11:21.147907 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:11:21.148707 (+ 800us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:11:21.148717 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:11:21.149325 (+ 608us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:11:21.149329 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:11:21.149429 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:11:21.149877 (+ 448us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:11:21.149891 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:11:21.151301 (+ 1410us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:11:21.154358 (+ 3057us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:11:21.154362 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:11:21.154364 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:11:21.154633 (+ 269us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:11:21.154964 (+ 331us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:11:21.154967 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:11:21.154969 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:11:21.155073 (+ 104us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:11:21.156034 (+ 961us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:11:21.156038 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:11:21.156144 (+ 106us) client_negotiation.cc:769] Sending connection context
0620 06:11:21.156259 (+ 115us) client_negotiation.cc:240] Negotiation successful
0620 06:11:21.156428 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":153}
I20260620 06:11:21.157461 1910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:11:21.145075 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40288 (local address 127.11.102.62:39415)
0620 06:11:21.145185 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0620 06:11:21.145190 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 06:11:21.145669 (+ 479us) server_negotiation.cc:407] Connection header received
0620 06:11:21.145870 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:11:21.145873 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:11:21.145922 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:11:21.145972 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:11:21.146911 (+ 939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:11:21.147393 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:11:21.148705 (+ 1312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:11:21.149310 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:11:21.149404 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:11:21.153258 (+ 3854us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:11:21.153264 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:11:21.153270 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:11:21.153299 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:11:21.154992 (+ 1693us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:11:21.155723 (+ 731us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:11:21.155726 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:11:21.155728 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:11:21.155809 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:11:21.156124 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:11:21.156128 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:11:21.156129 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:11:21.156452 (+ 323us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:11:21.156534 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0620 06:11:21.157142 (+ 608us) server_negotiation.cc:299] Negotiation successful
0620 06:11:21.157265 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":193,"server-negotiator.queue_time_us":46,"spinlock_wait_cycles":1408}
I20260620 06:11:21.157791 1909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:11:21.145743 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.102.60:42801 (local address 127.0.0.1:46084)
0620 06:11:21.145907 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0620 06:11:21.145919 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 06:11:21.145980 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:11:21.146949 (+ 969us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:11:21.146952 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:11:21.147144 (+ 192us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:11:21.147237 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:11:21.147243 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:11:21.148281 (+ 1038us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:11:21.148284 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:11:21.149078 (+ 794us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:11:21.149087 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:11:21.150050 (+ 963us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:11:21.150056 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:11:21.150148 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:11:21.150606 (+ 458us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:11:21.150619 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:11:21.152085 (+ 1466us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:11:21.155978 (+ 3893us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:11:21.155981 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:11:21.155983 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:11:21.156222 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:11:21.156518 (+ 296us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:11:21.156522 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:11:21.156524 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:11:21.156602 (+ 78us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:11:21.157360 (+ 758us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:11:21.157363 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:11:21.157447 (+ 84us) client_negotiation.cc:769] Sending connection context
0620 06:11:21.157522 (+ 75us) client_negotiation.cc:240] Negotiation successful
0620 06:11:21.157644 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":88,"mutex_wait_us":1}
I20260620 06:11:21.158344 2062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:11:21.145665 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46070 (local address 127.11.102.60:42801)
0620 06:11:21.146285 (+ 620us) server_negotiation.cc:206] Beginning negotiation
0620 06:11:21.146296 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 06:11:21.146311 (+ 15us) server_negotiation.cc:407] Connection header received
0620 06:11:21.146350 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:11:21.146355 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:11:21.146399 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:11:21.146499 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:11:21.146997 (+ 498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:11:21.147780 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:11:21.148831 (+ 1051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:11:21.149203 (+ 372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:11:21.149309 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:11:21.151452 (+ 2143us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:11:21.151456 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:11:21.151458 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:11:21.151489 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:11:21.154221 (+ 2732us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:11:21.154748 (+ 527us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:11:21.154753 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:11:21.154758 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:11:21.154863 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:11:21.155183 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:11:21.155189 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:11:21.155192 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:11:21.155870 (+ 678us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:11:21.155986 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0620 06:11:21.156942 (+ 956us) server_negotiation.cc:299] Negotiation successful
0620 06:11:21.157260 (+ 318us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":517,"spinlock_wait_cycles":10880,"thread_start_us":88,"threads_started":1}
I20260620 06:11:21.158331 2060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:11:21.140052 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46054 (local address 127.11.102.60:42801)
0620 06:11:21.140494 (+ 442us) server_negotiation.cc:206] Beginning negotiation
0620 06:11:21.140512 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0620 06:11:21.140555 (+ 43us) server_negotiation.cc:407] Connection header received
0620 06:11:21.141305 (+ 750us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:11:21.141329 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:11:21.141757 (+ 428us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:11:21.142135 (+ 378us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:11:21.142844 (+ 709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:11:21.144307 (+ 1463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:11:21.145565 (+ 1258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:11:21.146258 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:11:21.146419 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:11:21.150447 (+ 4028us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:11:21.150483 (+ 36us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:11:21.150494 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:11:21.150526 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:11:21.155534 (+ 5008us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:11:21.156113 (+ 579us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:11:21.156116 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:11:21.156118 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:11:21.156192 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:11:21.156492 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:11:21.156498 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:11:21.156503 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:11:21.156690 (+ 187us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:11:21.156854 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0620 06:11:21.157552 (+ 698us) server_negotiation.cc:299] Negotiation successful
0620 06:11:21.157672 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":294,"thread_start_us":107,"threads_started":1}
I20260620 06:11:21.158331 2061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:11:21.144862 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.102.62:39415 (local address 127.0.0.1:40288)
0620 06:11:21.145544 (+ 682us) negotiation.cc:107] Waiting for socket to connect
0620 06:11:21.145596 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0620 06:11:21.145749 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:11:21.146151 (+ 402us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:11:21.146157 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:11:21.146623 (+ 466us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:11:21.146746 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:11:21.146754 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:11:21.147509 (+ 755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:11:21.147511 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:11:21.148552 (+ 1041us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:11:21.148565 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:11:21.150447 (+ 1882us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:11:21.150452 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:11:21.150566 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:11:21.151117 (+ 551us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:11:21.151136 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:11:21.153057 (+ 1921us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:11:21.155095 (+ 2038us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:11:21.155109 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:11:21.155133 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:11:21.155597 (+ 464us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:11:21.155928 (+ 331us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:11:21.155931 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:11:21.155933 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:11:21.156019 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:11:21.156598 (+ 579us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:11:21.156604 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:11:21.156884 (+ 280us) client_negotiation.cc:769] Sending connection context
0620 06:11:21.157641 (+ 757us) client_negotiation.cc:240] Negotiation successful
0620 06:11:21.157801 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":456,"thread_start_us":128,"threads_started":1}
I20260620 06:11:21.159077 2063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:11:21.146154 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46084 (local address 127.11.102.60:42801)
0620 06:11:21.146721 (+ 567us) server_negotiation.cc:206] Beginning negotiation
0620 06:11:21.146729 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 06:11:21.146739 (+ 10us) server_negotiation.cc:407] Connection header received
0620 06:11:21.146768 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:11:21.146771 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:11:21.146803 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:11:21.147341 (+ 538us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:11:21.147443 (+ 102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:11:21.148163 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:11:21.149554 (+ 1391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:11:21.149927 (+ 373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:11:21.150037 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:11:21.152240 (+ 2203us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:11:21.152244 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:11:21.152248 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:11:21.152277 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:11:21.155855 (+ 3578us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:11:21.156338 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:11:21.156340 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:11:21.156342 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:11:21.156419 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:11:21.157058 (+ 639us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:11:21.157061 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:11:21.157063 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:11:21.157244 (+ 181us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:11:21.158825 (+ 1581us) server_negotiation.cc:1035] Waiting for connection context
0620 06:11:21.158884 (+ 59us) server_negotiation.cc:299] Negotiation successful
0620 06:11:21.158968 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":65,"server-negotiator.queue_time_us":505,"spinlock_wait_cycles":1920,"thread_start_us":86,"threads_started":1}
I20260620 06:11:21.160694 2057 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.11.102.61" port: 39357 } has no permanent_uuid. Determining permanent_uuid...
I20260620 06:11:21.162595 1906 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dccea233716e45e2aaea40678c2f6bf1: Bootstrap starting.
I20260620 06:11:21.162693 1981 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 51c2e75c02754412bb63721962c22346: Bootstrap starting.
I20260620 06:11:21.165201 1981 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 51c2e75c02754412bb63721962c22346: Neither blocks nor log segments found. Creating new log.
I20260620 06:11:21.165222 1906 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P dccea233716e45e2aaea40678c2f6bf1: Neither blocks nor log segments found. Creating new log.
I20260620 06:11:21.165943 1981 log.cc:824] T 00000000000000000000000000000000 P 51c2e75c02754412bb63721962c22346: Log is configured to *not* fsync() on all Append() calls
I20260620 06:11:21.165992 1906 log.cc:824] T 00000000000000000000000000000000 P dccea233716e45e2aaea40678c2f6bf1: Log is configured to *not* fsync() on all Append() calls
I20260620 06:11:21.168251 1906 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dccea233716e45e2aaea40678c2f6bf1: No bootstrap required, opened a new log
I20260620 06:11:21.168536 1981 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 51c2e75c02754412bb63721962c22346: No bootstrap required, opened a new log
Jun 20 06:11:21 dist-test-slave-9hm5 krb5kdc[1825](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935881, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.102.60@KRBTEST.COM for kudu/127.11.102.61@KRBTEST.COM
I20260620 06:11:21.172258 1981 raft_consensus.cc:348] T 00000000000000000000000000000000 P 51c2e75c02754412bb63721962c22346 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dccea233716e45e2aaea40678c2f6bf1" member_type: VOTER last_known_addr { host: "127.11.102.62" port: 39415 } } peers { permanent_uuid: "51c2e75c02754412bb63721962c22346" member_type: VOTER last_known_addr { host: "127.11.102.61" port: 39357 } } peers { permanent_uuid: "501d32ad709a4a42a12a3f712695adbf" member_type: VOTER last_known_addr { host: "127.11.102.60" port: 42801 } }
I20260620 06:11:21.172503 1981 raft_consensus.cc:729] T 00000000000000000000000000000000 P 51c2e75c02754412bb63721962c22346 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51c2e75c02754412bb63721962c22346, State: Initialized, Role: FOLLOWER
I20260620 06:11:21.172502 1906 raft_consensus.cc:348] T 00000000000000000000000000000000 P dccea233716e45e2aaea40678c2f6bf1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dccea233716e45e2aaea40678c2f6bf1" member_type: VOTER last_known_addr { host: "127.11.102.62" port: 39415 } } peers { permanent_uuid: "51c2e75c02754412bb63721962c22346" member_type: VOTER last_known_addr { host: "127.11.102.61" port: 39357 } } peers { permanent_uuid: "501d32ad709a4a42a12a3f712695adbf" member_type: VOTER last_known_addr { host: "127.11.102.60" port: 42801 } }
I20260620 06:11:21.172740 1906 raft_consensus.cc:729] T 00000000000000000000000000000000 P dccea233716e45e2aaea40678c2f6bf1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dccea233716e45e2aaea40678c2f6bf1, State: Initialized, Role: FOLLOWER
I20260620 06:11:21.172851 1981 consensus_queue.cc:260] T 00000000000000000000000000000000 P 51c2e75c02754412bb63721962c22346 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dccea233716e45e2aaea40678c2f6bf1" member_type: VOTER last_known_addr { host: "127.11.102.62" port: 39415 } } peers { permanent_uuid: "51c2e75c02754412bb63721962c22346" member_type: VOTER last_known_addr { host: "127.11.102.61" port: 39357 } } peers { permanent_uuid: "501d32ad709a4a42a12a3f712695adbf" member_type: VOTER last_known_addr { host: "127.11.102.60" port: 42801 } }
I20260620 06:11:21.173043 1906 consensus_queue.cc:260] T 00000000000000000000000000000000 P dccea233716e45e2aaea40678c2f6bf1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dccea233716e45e2aaea40678c2f6bf1" member_type: VOTER last_known_addr { host: "127.11.102.62" port: 39415 } } peers { permanent_uuid: "51c2e75c02754412bb63721962c22346" member_type: VOTER last_known_addr { host: "127.11.102.61" port: 39357 } } peers { permanent_uuid: "501d32ad709a4a42a12a3f712695adbf" member_type: VOTER last_known_addr { host: "127.11.102.60" port: 42801 } }
I20260620 06:11:21.173254 1987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:11:21.164098 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49812 (local address 127.11.102.61:39357)
0620 06:11:21.164221 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0620 06:11:21.164226 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 06:11:21.164279 (+ 53us) server_negotiation.cc:407] Connection header received
0620 06:11:21.164455 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:11:21.164458 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:11:21.164509 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:11:21.164594 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:11:21.165058 (+ 464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:11:21.165859 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:11:21.166790 (+ 931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:11:21.167044 (+ 254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:11:21.167166 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:11:21.169462 (+ 2296us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:11:21.169466 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:11:21.169469 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:11:21.169494 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:11:21.171252 (+ 1758us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:11:21.171815 (+ 563us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:11:21.171819 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:11:21.171821 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:11:21.171906 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:11:21.172217 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:11:21.172221 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:11:21.172223 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:11:21.172484 (+ 261us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:11:21.172563 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0620 06:11:21.172962 (+ 399us) server_negotiation.cc:299] Negotiation successful
0620 06:11:21.173091 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260620 06:11:21.173763 2061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:11:21.163887 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.102.61:39357 (local address 127.0.0.1:49812)
0620 06:11:21.164213 (+ 326us) negotiation.cc:107] Waiting for socket to connect
0620 06:11:21.164227 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 06:11:21.164339 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:11:21.164627 (+ 288us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:11:21.164631 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:11:21.164814 (+ 183us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:11:21.164926 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:11:21.164933 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:11:21.165986 (+ 1053us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:11:21.165990 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:11:21.166672 (+ 682us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:11:21.166681 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:11:21.167164 (+ 483us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:11:21.167168 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:11:21.167264 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:11:21.167838 (+ 574us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:11:21.167853 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:11:21.169309 (+ 1456us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:11:21.171375 (+ 2066us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:11:21.171379 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:11:21.171382 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:11:21.171687 (+ 305us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:11:21.172018 (+ 331us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:11:21.172021 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:11:21.172023 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:11:21.172104 (+ 81us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:11:21.172671 (+ 567us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:11:21.172674 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:11:21.172770 (+ 96us) client_negotiation.cc:769] Sending connection context
0620 06:11:21.173435 (+ 665us) client_negotiation.cc:240] Negotiation successful
0620 06:11:21.173571 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":259}
I20260620 06:11:21.174332 1981 sys_catalog.cc:564] T 00000000000000000000000000000000 P 51c2e75c02754412bb63721962c22346 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 06:11:21.174350 1906 sys_catalog.cc:564] T 00000000000000000000000000000000 P dccea233716e45e2aaea40678c2f6bf1 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 06:11:21.174435 2057 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.11.102.60" port: 42801 } has no permanent_uuid. Determining permanent_uuid...
W20260620 06:11:21.174858 1859 tablet.cc:2369] T 00000000000000000000000000000000 P dccea233716e45e2aaea40678c2f6bf1: Can't schedule compaction. Clean time has not been advanced past its initial value.
Jun 20 06:11:21 dist-test-slave-9hm5 krb5kdc[1825](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935880, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.102.62@KRBTEST.COM
W20260620 06:11:21.180043 2089 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 51c2e75c02754412bb63721962c22346: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 06:11:21.180121 2089 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
Jun 20 06:11:21 dist-test-slave-9hm5 krb5kdc[1825](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935881, etypes {rep=17 tkt=17 ses=17}, kudu/127.11.102.60@KRBTEST.COM for kudu/127.11.102.60@KRBTEST.COM
I20260620 06:11:21.183986 2063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:11:21.175142 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46096 (local address 127.11.102.60:42801)
0620 06:11:21.175247 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0620 06:11:21.175254 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 06:11:21.175385 (+ 131us) server_negotiation.cc:407] Connection header received
0620 06:11:21.175545 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:11:21.175549 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:11:21.175594 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:11:21.175677 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:11:21.176192 (+ 515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:11:21.176964 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:11:21.177909 (+ 945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:11:21.178227 (+ 318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:11:21.178301 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:11:21.180653 (+ 2352us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:11:21.180656 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:11:21.180658 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:11:21.180677 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:11:21.182261 (+ 1584us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:11:21.182692 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:11:21.182695 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:11:21.182697 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:11:21.182769 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:11:21.183039 (+ 270us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:11:21.183042 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:11:21.183043 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:11:21.183237 (+ 194us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:11:21.183329 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0620 06:11:21.183705 (+ 376us) server_negotiation.cc:299] Negotiation successful
0620 06:11:21.183828 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260620 06:11:21.184369 2061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:11:21.174940 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.11.102.60:42801 (local address 127.0.0.1:46096)
0620 06:11:21.175322 (+ 382us) negotiation.cc:107] Waiting for socket to connect
0620 06:11:21.175338 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 06:11:21.175440 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:11:21.175821 (+ 381us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:11:21.175823 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:11:21.175989 (+ 166us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:11:21.176067 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:11:21.176074 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:11:21.177082 (+ 1008us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:11:21.177085 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:11:21.177794 (+ 709us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:11:21.177804 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:11:21.178419 (+ 615us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:11:21.178422 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:11:21.178488 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:11:21.179071 (+ 583us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:11:21.179085 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:11:21.180529 (+ 1444us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:11:21.182370 (+ 1841us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:11:21.182372 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:11:21.182374 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:11:21.182591 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:11:21.182874 (+ 283us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:11:21.182876 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:11:21.182878 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:11:21.182954 (+ 76us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:11:21.183364 (+ 410us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:11:21.183367 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:11:21.183447 (+ 80us) client_negotiation.cc:769] Sending connection context
0620 06:11:21.183543 (+ 96us) client_negotiation.cc:240] Negotiation successful
0620 06:11:21.184223 (+ 680us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":328}
I20260620 06:11:21.184743 1910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:11:21.164191 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40296 (local address 127.11.102.62:39415)
0620 06:11:21.164286 (+ 95us) server_negotiation.cc:206] Beginning negotiation
0620 06:11:21.164290 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 06:11:21.166577 (+ 2287us) server_negotiation.cc:407] Connection header received
0620 06:11:21.166768 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:11:21.166771 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:11:21.166823 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:11:21.166877 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:11:21.172735 (+ 5858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:11:21.173449 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:11:21.176339 (+ 2890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:11:21.176707 (+ 368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:11:21.176792 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:11:21.180360 (+ 3568us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:11:21.180363 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:11:21.180366 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:11:21.180390 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:11:21.182482 (+ 2092us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:11:21.183000 (+ 518us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:11:21.183004 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:11:21.183005 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:11:21.183093 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:11:21.183412 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:11:21.183416 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:11:21.183432 (+ 16us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:11:21.183675 (+ 243us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:11:21.183825 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0620 06:11:21.184459 (+ 634us) server_negotiation.cc:299] Negotiation successful
0620 06:11:21.184585 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260620 06:11:21.187440 2057 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 501d32ad709a4a42a12a3f712695adbf: Bootstrap starting.
I20260620 06:11:21.184304 1913 master_runner.cc:186] Error getting master registration for 127.11.102.60:42801: OK, instance_id { permanent_uuid: "501d32ad709a4a42a12a3f712695adbf" instance_seqno: 1781935881132307 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260620 06:11:21.187772 1913 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260620 06:11:21.188692 2067 sys_catalog.cc:455] T 00000000000000000000000000000000 P dccea233716e45e2aaea40678c2f6bf1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dccea233716e45e2aaea40678c2f6bf1" member_type: VOTER last_known_addr { host: "127.11.102.62" port: 39415 } } peers { permanent_uuid: "51c2e75c02754412bb63721962c22346" member_type: VOTER last_known_addr { host: "127.11.102.61" port: 39357 } } peers { permanent_uuid: "501d32ad709a4a42a12a3f712695adbf" member_type: VOTER last_known_addr { host: "127.11.102.60" port: 42801 } } }
I20260620 06:11:21.188858 2067 sys_catalog.cc:458] T 00000000000000000000000000000000 P dccea233716e45e2aaea40678c2f6bf1 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 06:11:21.189065 2066 sys_catalog.cc:455] T 00000000000000000000000000000000 P 51c2e75c02754412bb63721962c22346 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dccea233716e45e2aaea40678c2f6bf1" member_type: VOTER last_known_addr { host: "127.11.102.62" port: 39415 } } peers { permanent_uuid: "51c2e75c02754412bb63721962c22346" member_type: VOTER last_known_addr { host: "127.11.102.61" port: 39357 } } peers { permanent_uuid: "501d32ad709a4a42a12a3f712695adbf" member_type: VOTER last_known_addr { host: "127.11.102.60" port: 42801 } } }
I20260620 06:11:21.189191 2066 sys_catalog.cc:458] T 00000000000000000000000000000000 P 51c2e75c02754412bb63721962c22346 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 06:11:21.192073 2057 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 501d32ad709a4a42a12a3f712695adbf: Neither blocks nor log segments found. Creating new log.
W20260620 06:11:21.192989 2086 catalog_manager.cc:1558] T 00000000000000000000000000000000 P dccea233716e45e2aaea40678c2f6bf1: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 06:11:21.193073 2086 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 06:11:21.193010 2057 log.cc:824] T 00000000000000000000000000000000 P 501d32ad709a4a42a12a3f712695adbf: Log is configured to *not* fsync() on all Append() calls
I20260620 06:11:21.189710 1838 master_runner.cc:186] Error getting master registration for 127.11.102.60:42801: OK, instance_id { permanent_uuid: "501d32ad709a4a42a12a3f712695adbf" instance_seqno: 1781935881132307 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260620 06:11:21.193663 1838 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260620 06:11:21.195523 2057 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 501d32ad709a4a42a12a3f712695adbf: No bootstrap required, opened a new log
Jun 20 06:11:21 dist-test-slave-9hm5 krb5kdc[1825](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935880, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.11.102.61@KRBTEST.COM
I20260620 06:11:21.200584 2057 raft_consensus.cc:348] T 00000000000000000000000000000000 P 501d32ad709a4a42a12a3f712695adbf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dccea233716e45e2aaea40678c2f6bf1" member_type: VOTER last_known_addr { host: "127.11.102.62" port: 39415 } } peers { permanent_uuid: "51c2e75c02754412bb63721962c22346" member_type: VOTER last_known_addr { host: "127.11.102.61" port: 39357 } } peers { permanent_uuid: "501d32ad709a4a42a12a3f712695adbf" member_type: VOTER last_known_addr { host: "127.11.102.60" port: 42801 } }
I20260620 06:11:21.200937 2057 raft_consensus.cc:729] T 00000000000000000000000000000000 P 501d32ad709a4a42a12a3f712695adbf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 501d32ad709a4a42a12a3f712695adbf, State: Initialized, Role: FOLLOWER
I20260620 06:11:21.201352 2057 consensus_queue.cc:260] T 00000000000000000000000000000000 P 501d32ad709a4a42a12a3f712695adbf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dccea233716e45e2aaea40678c2f6bf1" member_type: VOTER last_known_addr { host: "127.11.102.62" port: 39415 } } peers { permanent_uuid: "51c2e75c02754412bb63721962c22346" member_type: VOTER last_known_addr { host: "127.11.102.61" port: 39357 } } peers { permanent_uuid: "501d32ad709a4a42a12a3f712695adbf" member_type: VOTER last_known_addr { host: "127.11.102.60" port: 42801 } }
I20260620 06:11:21.202996 2057 sys_catalog.cc:564] T 00000000000000000000000000000000 P 501d32ad709a4a42a12a3f712695adbf [sys.catalog]: configured and running, proceeding with master startup.
I20260620 06:11:21.204010 1987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:11:21.190701 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49822 (local address 127.11.102.61:39357)
0620 06:11:21.190807 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0620 06:11:21.190811 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 06:11:21.193327 (+ 2516us) server_negotiation.cc:407] Connection header received
0620 06:11:21.193517 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:11:21.193520 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:11:21.193568 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:11:21.193615 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:11:21.194496 (+ 881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:11:21.195286 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:11:21.196447 (+ 1161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:11:21.196803 (+ 356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:11:21.196876 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:11:21.200055 (+ 3179us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:11:21.200060 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:11:21.200062 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:11:21.200094 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:11:21.201922 (+ 1828us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:11:21.202527 (+ 605us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:11:21.202531 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:11:21.202533 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:11:21.202619 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:11:21.202911 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:11:21.202914 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:11:21.202916 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:11:21.203132 (+ 216us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:11:21.203326 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0620 06:11:21.203704 (+ 378us) server_negotiation.cc:299] Negotiation successful
0620 06:11:21.203846 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260620 06:11:21.204746 2093 sys_catalog.cc:455] T 00000000000000000000000000000000 P 501d32ad709a4a42a12a3f712695adbf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dccea233716e45e2aaea40678c2f6bf1" member_type: VOTER last_known_addr { host: "127.11.102.62" port: 39415 } } peers { permanent_uuid: "51c2e75c02754412bb63721962c22346" member_type: VOTER last_known_addr { host: "127.11.102.61" port: 39357 } } peers { permanent_uuid: "501d32ad709a4a42a12a3f712695adbf" member_type: VOTER last_known_addr { host: "127.11.102.60" port: 42801 } } }
I20260620 06:11:21.204900 2093 sys_catalog.cc:458] T 00000000000000000000000000000000 P 501d32ad709a4a42a12a3f712695adbf [sys.catalog]: This master's current role is: FOLLOWER
W20260620 06:11:21.209160 2104 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 501d32ad709a4a42a12a3f712695adbf: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 06:11:21.209235 2104 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 06:11:21.209719 11672 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 06:11:21.211239 1880 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "dccea233716e45e2aaea40678c2f6bf1"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40296
I20260620 06:11:21.211493 1880 raft_consensus.cc:482] T 00000000000000000000000000000000 P dccea233716e45e2aaea40678c2f6bf1 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260620 06:11:21.211596 1880 raft_consensus.cc:3037] T 00000000000000000000000000000000 P dccea233716e45e2aaea40678c2f6bf1 [term 0 FOLLOWER]: Advancing to term 1
I20260620 06:11:21.212957 1880 raft_consensus.cc:504] T 00000000000000000000000000000000 P dccea233716e45e2aaea40678c2f6bf1 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dccea233716e45e2aaea40678c2f6bf1" member_type: VOTER last_known_addr { host: "127.11.102.62" port: 39415 } } peers { permanent_uuid: "51c2e75c02754412bb63721962c22346" member_type: VOTER last_known_addr { host: "127.11.102.61" port: 39357 } } peers { permanent_uuid: "501d32ad709a4a42a12a3f712695adbf" member_type: VOTER last_known_addr { host: "127.11.102.60" port: 42801 } }
I20260620 06:11:21.214586 2031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "dccea233716e45e2aaea40678c2f6bf1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "501d32ad709a4a42a12a3f712695adbf"
I20260620 06:11:21.214811 2031 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 501d32ad709a4a42a12a3f712695adbf [term 0 FOLLOWER]: Advancing to term 1
I20260620 06:11:21.216086 1955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "dccea233716e45e2aaea40678c2f6bf1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "51c2e75c02754412bb63721962c22346"
I20260620 06:11:21.216311 1955 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 51c2e75c02754412bb63721962c22346 [term 0 FOLLOWER]: Advancing to term 1
I20260620 06:11:21.218050 1955 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 51c2e75c02754412bb63721962c22346 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dccea233716e45e2aaea40678c2f6bf1 in term 1.
I20260620 06:11:21.218582 1855 leader_election.cc:302] T 00000000000000000000000000000000 P dccea233716e45e2aaea40678c2f6bf1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 51c2e75c02754412bb63721962c22346, dccea233716e45e2aaea40678c2f6bf1; no voters:
I20260620 06:11:21.218892 2067 raft_consensus.cc:2781] T 00000000000000000000000000000000 P dccea233716e45e2aaea40678c2f6bf1 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 06:11:21.219112 2067 raft_consensus.cc:686] T 00000000000000000000000000000000 P dccea233716e45e2aaea40678c2f6bf1 [term 1 LEADER]: Becoming Leader. State: Replica: dccea233716e45e2aaea40678c2f6bf1, State: Running, Role: LEADER
I20260620 06:11:21.219467 2067 consensus_queue.cc:237] T 00000000000000000000000000000000 P dccea233716e45e2aaea40678c2f6bf1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dccea233716e45e2aaea40678c2f6bf1" member_type: VOTER last_known_addr { host: "127.11.102.62" port: 39415 } } peers { permanent_uuid: "51c2e75c02754412bb63721962c22346" member_type: VOTER last_known_addr { host: "127.11.102.61" port: 39357 } } peers { permanent_uuid: "501d32ad709a4a42a12a3f712695adbf" member_type: VOTER last_known_addr { host: "127.11.102.60" port: 42801 } }
I20260620 06:11:21.221333 2067 sys_catalog.cc:455] T 00000000000000000000000000000000 P dccea233716e45e2aaea40678c2f6bf1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader dccea233716e45e2aaea40678c2f6bf1. Latest consensus state: current_term: 1 leader_uuid: "dccea233716e45e2aaea40678c2f6bf1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dccea233716e45e2aaea40678c2f6bf1" member_type: VOTER last_known_addr { host: "127.11.102.62" port: 39415 } } peers { permanent_uuid: "51c2e75c02754412bb63721962c22346" member_type: VOTER last_known_addr { host: "127.11.102.61" port: 39357 } } peers { permanent_uuid: "501d32ad709a4a42a12a3f712695adbf" member_type: VOTER last_known_addr { host: "127.11.102.60" port: 42801 } } }
I20260620 06:11:21.221452 2067 sys_catalog.cc:458] T 00000000000000000000000000000000 P dccea233716e45e2aaea40678c2f6bf1 [sys.catalog]: This master's current role is: LEADER
I20260620 06:11:21.223713 2031 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 501d32ad709a4a42a12a3f712695adbf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dccea233716e45e2aaea40678c2f6bf1 in term 1.
I20260620 06:11:21.224651 1989 master_runner.cc:232] Existing masters have differing terms: 0 vs 1
I20260620 06:11:21.224777 1989 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260620 06:11:21.224889 1880 leader_election.cc:288] T 00000000000000000000000000000000 P dccea233716e45e2aaea40678c2f6bf1 [CANDIDATE]: Term 1 election: Requested vote from peers 51c2e75c02754412bb63721962c22346 (127.11.102.61:39357), 501d32ad709a4a42a12a3f712695adbf (127.11.102.60:42801)
I20260620 06:11:21.227098 2108 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 06:11:21.228147 2108 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 06:11:21.239939 2031 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 501d32ad709a4a42a12a3f712695adbf [term 1 FOLLOWER]: Refusing update from remote peer dccea233716e45e2aaea40678c2f6bf1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 06:11:21.241070 2106 consensus_queue.cc:1035] T 00000000000000000000000000000000 P dccea233716e45e2aaea40678c2f6bf1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "501d32ad709a4a42a12a3f712695adbf" member_type: VOTER last_known_addr { host: "127.11.102.60" port: 42801 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 06:11:21.245270 2093 sys_catalog.cc:455] T 00000000000000000000000000000000 P 501d32ad709a4a42a12a3f712695adbf [sys.catalog]: SysCatalogTable state changed. Reason: New leader dccea233716e45e2aaea40678c2f6bf1. Latest consensus state: current_term: 1 leader_uuid: "dccea233716e45e2aaea40678c2f6bf1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dccea233716e45e2aaea40678c2f6bf1" member_type: VOTER last_known_addr { host: "127.11.102.62" port: 39415 } } peers { permanent_uuid: "51c2e75c02754412bb63721962c22346" member_type: VOTER last_known_addr { host: "127.11.102.61" port: 39357 } } peers { permanent_uuid: "501d32ad709a4a42a12a3f712695adbf" member_type: VOTER last_known_addr { host: "127.11.102.60" port: 42801 } } }
I20260620 06:11:21.245412 2093 sys_catalog.cc:458] T 00000000000000000000000000000000 P 501d32ad709a4a42a12a3f712695adbf [sys.catalog]: This master's current role is: FOLLOWER
I20260620 06:11:21.247503 1955 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 51c2e75c02754412bb63721962c22346 [term 1 FOLLOWER]: Refusing update from remote peer dccea233716e45e2aaea40678c2f6bf1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 06:11:21.248298 2067 consensus_queue.cc:1035] T 00000000000000000000000000000000 P dccea233716e45e2aaea40678c2f6bf1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "51c2e75c02754412bb63721962c22346" member_type: VOTER last_known_addr { host: "127.11.102.61" port: 39357 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 06:11:21.249471 2063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:11:21.228767 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46098 (local address 127.11.102.60:42801)
0620 06:11:21.228883 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0620 06:11:21.228887 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 06:11:21.235915 (+ 7028us) server_negotiation.cc:407] Connection header received
0620 06:11:21.236117 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:11:21.236120 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:11:21.236172 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:11:21.236225 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:11:21.237280 (+ 1055us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:11:21.237988 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:11:21.239462 (+ 1474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:11:21.239777 (+ 315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:11:21.239863 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:11:21.245689 (+ 5826us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:11:21.245693 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:11:21.245696 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:11:21.245724 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:11:21.247512 (+ 1788us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:11:21.248070 (+ 558us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:11:21.248076 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:11:21.248081 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:11:21.248173 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:11:21.248497 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:11:21.248501 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:11:21.248502 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:11:21.248730 (+ 228us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:11:21.248870 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0620 06:11:21.249209 (+ 339us) server_negotiation.cc:299] Negotiation successful
0620 06:11:21.249328 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260620 06:11:21.253448 2066 sys_catalog.cc:455] T 00000000000000000000000000000000 P 51c2e75c02754412bb63721962c22346 [sys.catalog]: SysCatalogTable state changed. Reason: New leader dccea233716e45e2aaea40678c2f6bf1. Latest consensus state: current_term: 1 leader_uuid: "dccea233716e45e2aaea40678c2f6bf1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dccea233716e45e2aaea40678c2f6bf1" member_type: VOTER last_known_addr { host: "127.11.102.62" port: 39415 } } peers { permanent_uuid: "51c2e75c02754412bb63721962c22346" member_type: VOTER last_known_addr { host: "127.11.102.61" port: 39357 } } peers { permanent_uuid: "501d32ad709a4a42a12a3f712695adbf" member_type: VOTER last_known_addr { host: "127.11.102.60" port: 42801 } } }
I20260620 06:11:21.253574 2066 sys_catalog.cc:458] T 00000000000000000000000000000000 P 51c2e75c02754412bb63721962c22346 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 06:11:21.267391 2106 sys_catalog.cc:455] T 00000000000000000000000000000000 P dccea233716e45e2aaea40678c2f6bf1 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "dccea233716e45e2aaea40678c2f6bf1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dccea233716e45e2aaea40678c2f6bf1" member_type: VOTER last_known_addr { host: "127.11.102.62" port: 39415 } } peers { permanent_uuid: "51c2e75c02754412bb63721962c22346" member_type: VOTER last_known_addr { host: "127.11.102.61" port: 39357 } } peers { permanent_uuid: "501d32ad709a4a42a12a3f712695adbf" member_type: VOTER last_known_addr { host: "127.11.102.60" port: 42801 } } }
I20260620 06:11:21.267554 2106 sys_catalog.cc:458] T 00000000000000000000000000000000 P dccea233716e45e2aaea40678c2f6bf1 [sys.catalog]: This master's current role is: LEADER
I20260620 06:11:21.269901 2066 sys_catalog.cc:455] T 00000000000000000000000000000000 P 51c2e75c02754412bb63721962c22346 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "dccea233716e45e2aaea40678c2f6bf1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dccea233716e45e2aaea40678c2f6bf1" member_type: VOTER last_known_addr { host: "127.11.102.62" port: 39415 } } peers { permanent_uuid: "51c2e75c02754412bb63721962c22346" member_type: VOTER last_known_addr { host: "127.11.102.61" port: 39357 } } peers { permanent_uuid: "501d32ad709a4a42a12a3f712695adbf" member_type: VOTER last_known_addr { host: "127.11.102.60" port: 42801 } } }
I20260620 06:11:21.270061 2066 sys_catalog.cc:458] T 00000000000000000000000000000000 P 51c2e75c02754412bb63721962c22346 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 06:11:21.270401 1987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:11:21.232224 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49832 (local address 127.11.102.61:39357)
0620 06:11:21.232346 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0620 06:11:21.232350 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 06:11:21.251090 (+ 18740us) server_negotiation.cc:407] Connection header received
0620 06:11:21.251270 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:11:21.251273 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:11:21.251325 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:11:21.251376 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:11:21.252241 (+ 865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:11:21.252960 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:11:21.263755 (+ 10795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:11:21.264114 (+ 359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:11:21.264204 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:11:21.266431 (+ 2227us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:11:21.266435 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:11:21.266438 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:11:21.266465 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:11:21.268468 (+ 2003us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:11:21.268975 (+ 507us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:11:21.268978 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:11:21.268981 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:11:21.269066 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:11:21.269377 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:11:21.269381 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:11:21.269383 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:11:21.269603 (+ 220us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:11:21.269722 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0620 06:11:21.270093 (+ 371us) server_negotiation.cc:299] Negotiation successful
0620 06:11:21.270221 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260620 06:11:21.272063 2114 mvcc.cc:205] Tried to move back new op lower bound from 7298809369549381632 to 7298809369478557696. Current Snapshot: MvccSnapshot[applied={T|T < 7298809369549381632}]
I20260620 06:11:21.273970 2067 sys_catalog.cc:455] T 00000000000000000000000000000000 P dccea233716e45e2aaea40678c2f6bf1 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "dccea233716e45e2aaea40678c2f6bf1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dccea233716e45e2aaea40678c2f6bf1" member_type: VOTER last_known_addr { host: "127.11.102.62" port: 39415 } } peers { permanent_uuid: "51c2e75c02754412bb63721962c22346" member_type: VOTER last_known_addr { host: "127.11.102.61" port: 39357 } } peers { permanent_uuid: "501d32ad709a4a42a12a3f712695adbf" member_type: VOTER last_known_addr { host: "127.11.102.60" port: 42801 } } }
I20260620 06:11:21.274097 2067 sys_catalog.cc:458] T 00000000000000000000000000000000 P dccea233716e45e2aaea40678c2f6bf1 [sys.catalog]: This master's current role is: LEADER
I20260620 06:11:21.275527 2093 sys_catalog.cc:455] T 00000000000000000000000000000000 P 501d32ad709a4a42a12a3f712695adbf [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "dccea233716e45e2aaea40678c2f6bf1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dccea233716e45e2aaea40678c2f6bf1" member_type: VOTER last_known_addr { host: "127.11.102.62" port: 39415 } } peers { permanent_uuid: "51c2e75c02754412bb63721962c22346" member_type: VOTER last_known_addr { host: "127.11.102.61" port: 39357 } } peers { permanent_uuid: "501d32ad709a4a42a12a3f712695adbf" member_type: VOTER last_known_addr { host: "127.11.102.60" port: 42801 } } }
I20260620 06:11:21.275678 2093 sys_catalog.cc:458] T 00000000000000000000000000000000 P 501d32ad709a4a42a12a3f712695adbf [sys.catalog]: This master's current role is: FOLLOWER
I20260620 06:11:21.277096 2119 mvcc.cc:205] Tried to move back new op lower bound from 7298809369549381632 to 7298809369478557696. Current Snapshot: MvccSnapshot[applied={T|T < 7298809369549381632}]
I20260620 06:11:21.277300 2117 mvcc.cc:205] Tried to move back new op lower bound from 7298809369549381632 to 7298809369478557696. Current Snapshot: MvccSnapshot[applied={T|T < 7298809369549381632}]
I20260620 06:11:21.278627 1910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:11:21.231556 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40306 (local address 127.11.102.62:39415)
0620 06:11:21.231682 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0620 06:11:21.231686 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 06:11:21.238425 (+ 6739us) server_negotiation.cc:407] Connection header received
0620 06:11:21.238604 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:11:21.238608 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:11:21.238655 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:11:21.238706 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:11:21.254020 (+ 15314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:11:21.254817 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:11:21.256068 (+ 1251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:11:21.256418 (+ 350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:11:21.256490 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:11:21.264709 (+ 8219us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:11:21.264715 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:11:21.264720 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:11:21.264755 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:11:21.268845 (+ 4090us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:11:21.276590 (+ 7745us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:11:21.276597 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:11:21.276600 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:11:21.276712 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:11:21.277468 (+ 756us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:11:21.277472 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:11:21.277474 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:11:21.277718 (+ 244us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:11:21.277849 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0620 06:11:21.278330 (+ 481us) server_negotiation.cc:299] Negotiation successful
0620 06:11:21.278466 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260620 06:11:21.287179 2108 catalog_manager.cc:1347] Generated new cluster ID: 7e4845424f7646818479afe86d290df9
I20260620 06:11:21.287267 2108 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 06:11:21.339696 2108 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 06:11:21.341150 2108 catalog_manager.cc:1504] Loading token signing keys...
I20260620 06:11:21.367250 2108 catalog_manager.cc:5868] T 00000000000000000000000000000000 P dccea233716e45e2aaea40678c2f6bf1: Generated new TSK 0
I20260620 06:11:21.368537 2108 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260620 06:11:21.403676 1987 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 06:11:21.401891 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49840 (local address 127.11.102.61:39357)
0620 06:11:21.402036 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0620 06:11:21.402041 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 06:11:21.402665 (+ 624us) server_negotiation.cc:407] Connection header received
0620 06:11:21.402855 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:11:21.402859 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:11:21.402907 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:11:21.402955 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:11:21.403511 (+ 556us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49840: BlockingRecv error: recv got EOF from 127.0.0.1:49840 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
W20260620 06:11:21.403676 1910 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 06:11:21.401471 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40316 (local address 127.11.102.62:39415)
0620 06:11:21.401594 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0620 06:11:21.401599 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 06:11:21.402522 (+ 923us) server_negotiation.cc:407] Connection header received
0620 06:11:21.402748 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:11:21.402752 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:11:21.402806 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:11:21.402860 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:11:21.403521 (+ 661us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40316: BlockingRecv error: recv got EOF from 127.0.0.1:40316 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
W20260620 06:11:21.404943 2063 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 06:11:21.402405 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46114 (local address 127.11.102.60:42801)
0620 06:11:21.402517 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0620 06:11:21.402521 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 06:11:21.404071 (+ 1550us) server_negotiation.cc:407] Connection header received
0620 06:11:21.404231 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:11:21.404234 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:11:21.404280 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:11:21.404328 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:11:21.404827 (+ 499us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46114: BlockingRecv error: recv got EOF from 127.0.0.1:46114 (error 108)
Metrics: {"server-negotiator.queue_time_us":32}
W20260620 06:11:21.408608 1987 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 06:11:21.407306 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49842 (local address 127.11.102.61:39357)
0620 06:11:21.407411 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0620 06:11:21.407415 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 06:11:21.407700 (+ 285us) server_negotiation.cc:407] Connection header received
0620 06:11:21.407859 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:11:21.407862 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:11:21.407939 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:11:21.408023 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:11:21.408479 (+ 456us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49842: BlockingRecv error: recv got EOF from 127.0.0.1:49842 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
W20260620 06:11:21.416801 1987 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 06:11:21.412352 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49856 (local address 127.11.102.61:39357)
0620 06:11:21.412478 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0620 06:11:21.412482 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 06:11:21.415946 (+ 3464us) server_negotiation.cc:407] Connection header received
0620 06:11:21.416077 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:11:21.416081 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:11:21.416138 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:11:21.416224 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:11:21.416682 (+ 458us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49856: BlockingRecv error: recv got EOF from 127.0.0.1:49856 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
W20260620 06:11:21.421624 1987 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 06:11:21.420632 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49866 (local address 127.11.102.61:39357)
0620 06:11:21.420741 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0620 06:11:21.420745 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 06:11:21.420888 (+ 143us) server_negotiation.cc:407] Connection header received
0620 06:11:21.420969 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:11:21.420973 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:11:21.421021 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:11:21.421105 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:11:21.421514 (+ 409us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49866: BlockingRecv error: recv got EOF from 127.0.0.1:49866 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
W20260620 06:11:21.428397 1987 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 06:11:21.427308 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49870 (local address 127.11.102.61:39357)
0620 06:11:21.427490 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0620 06:11:21.427494 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 06:11:21.427576 (+ 82us) server_negotiation.cc:407] Connection header received
0620 06:11:21.427750 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:11:21.427753 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:11:21.427803 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:11:21.427890 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:11:21.428314 (+ 424us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49870: BlockingRecv error: recv got EOF from 127.0.0.1:49870 (error 108)
Metrics: {"server-negotiator.queue_time_us":114}
W20260620 06:11:21.439191 1987 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 06:11:21.438130 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49876 (local address 127.11.102.61:39357)
0620 06:11:21.438282 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0620 06:11:21.438286 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 06:11:21.438401 (+ 115us) server_negotiation.cc:407] Connection header received
0620 06:11:21.438554 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:11:21.438557 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:11:21.438607 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:11:21.438694 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:11:21.439097 (+ 403us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49876: BlockingRecv error: recv got EOF from 127.0.0.1:49876 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
W20260620 06:11:21.458420 1987 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 06:11:21.457198 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49884 (local address 127.11.102.61:39357)
0620 06:11:21.457360 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0620 06:11:21.457365 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 06:11:21.457521 (+ 156us) server_negotiation.cc:407] Connection header received
0620 06:11:21.457697 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:11:21.457700 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:11:21.457768 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:11:21.457879 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:11:21.458309 (+ 430us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49884: BlockingRecv error: recv got EOF from 127.0.0.1:49884 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
W20260620 06:11:21.493966 1987 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 06:11:21.492783 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49886 (local address 127.11.102.61:39357)
0620 06:11:21.492966 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0620 06:11:21.492969 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 06:11:21.493032 (+ 63us) server_negotiation.cc:407] Connection header received
0620 06:11:21.493160 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:11:21.493163 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:11:21.493213 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:11:21.493292 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:11:21.493838 (+ 546us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49886: BlockingRecv error: recv got EOF from 127.0.0.1:49886 (error 108)
Metrics: {"server-negotiator.queue_time_us":87}
W20260620 06:11:21.561286 1987 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 06:11:21.560034 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49892 (local address 127.11.102.61:39357)
0620 06:11:21.560182 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0620 06:11:21.560187 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 06:11:21.560396 (+ 209us) server_negotiation.cc:407] Connection header received
0620 06:11:21.560564 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:11:21.560567 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:11:21.560615 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:11:21.560710 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:11:21.561184 (+ 474us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49892: BlockingRecv error: recv got EOF from 127.0.0.1:49892 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
W20260620 06:11:21.692744 1987 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 06:11:21.691471 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49898 (local address 127.11.102.61:39357)
0620 06:11:21.691607 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0620 06:11:21.691630 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0620 06:11:21.691821 (+ 191us) server_negotiation.cc:407] Connection header received
0620 06:11:21.692001 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:11:21.692004 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:11:21.692055 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:11:21.692145 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:11:21.692606 (+ 461us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49898: BlockingRecv error: recv got EOF from 127.0.0.1:49898 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
W20260620 06:11:21.952451 1987 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 06:11:21.951329 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49900 (local address 127.11.102.61:39357)
0620 06:11:21.951497 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0620 06:11:21.951501 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 06:11:21.951518 (+ 17us) server_negotiation.cc:407] Connection header received
0620 06:11:21.951704 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:11:21.951706 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:11:21.951761 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:11:21.951852 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:11:21.952306 (+ 454us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49900: BlockingRecv error: recv got EOF from 127.0.0.1:49900 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
I20260620 06:11:22.181362 2089 catalog_manager.cc:1259] Loaded cluster ID: 7e4845424f7646818479afe86d290df9
I20260620 06:11:22.181474 2089 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 51c2e75c02754412bb63721962c22346: loading cluster ID for follower catalog manager: success
I20260620 06:11:22.183147 2089 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 51c2e75c02754412bb63721962c22346: acquiring CA information for follower catalog manager: success
I20260620 06:11:22.184324 2089 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 51c2e75c02754412bb63721962c22346: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 06:11:22.210424 2104 catalog_manager.cc:1259] Loaded cluster ID: 7e4845424f7646818479afe86d290df9
I20260620 06:11:22.210500 2104 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 501d32ad709a4a42a12a3f712695adbf: loading cluster ID for follower catalog manager: success
I20260620 06:11:22.212167 2104 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 501d32ad709a4a42a12a3f712695adbf: acquiring CA information for follower catalog manager: success
I20260620 06:11:22.213282 2104 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 501d32ad709a4a42a12a3f712695adbf: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 06:11:22.470754 2188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:11:22.466657 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49916 (local address 127.11.102.61:39357)
0620 06:11:22.466957 (+ 300us) server_negotiation.cc:206] Beginning negotiation
0620 06:11:22.466963 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 06:11:22.466981 (+ 18us) server_negotiation.cc:407] Connection header received
0620 06:11:22.467131 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:11:22.467135 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:11:22.467281 (+ 146us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:11:22.467412 (+ 131us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 06:11:22.467726 (+ 314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:11:22.468277 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:11:22.469170 (+ 893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:11:22.469418 (+ 248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:11:22.469542 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:11:22.469874 (+ 332us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 06:11:22.470317 (+ 443us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 06:11:22.470468 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0620 06:11:22.470514 (+ 46us) server_negotiation.cc:299] Negotiation successful
0620 06:11:22.470574 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":213,"thread_start_us":103,"threads_started":1}
I20260620 06:11:22.478232 2194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:11:22.473957 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46130 (local address 127.11.102.60:42801)
0620 06:11:22.474219 (+ 262us) server_negotiation.cc:206] Beginning negotiation
0620 06:11:22.474222 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 06:11:22.474237 (+ 15us) server_negotiation.cc:407] Connection header received
0620 06:11:22.474403 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:11:22.474406 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:11:22.474522 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:11:22.474623 (+ 101us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 06:11:22.474990 (+ 367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:11:22.475664 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:11:22.476554 (+ 890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:11:22.476896 (+ 342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:11:22.477020 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:11:22.477354 (+ 334us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 06:11:22.477787 (+ 433us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 06:11:22.477937 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0620 06:11:22.477981 (+ 44us) server_negotiation.cc:299] Negotiation successful
0620 06:11:22.478040 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":189,"thread_start_us":79,"threads_started":1}
I20260620 06:11:22.485714 2188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:11:22.482016 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49928 (local address 127.11.102.61:39357)
0620 06:11:22.482176 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0620 06:11:22.482181 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 06:11:22.482197 (+ 16us) server_negotiation.cc:407] Connection header received
0620 06:11:22.482296 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:11:22.482306 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:11:22.482349 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:11:22.482434 (+ 85us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 06:11:22.482814 (+ 380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:11:22.483331 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:11:22.484441 (+ 1110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:11:22.484771 (+ 330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:11:22.484890 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:11:22.485140 (+ 250us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 06:11:22.485237 (+ 97us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 06:11:22.485291 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0620 06:11:22.485500 (+ 209us) server_negotiation.cc:299] Negotiation successful
0620 06:11:22.485569 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260620 06:11:22.485931 2194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:11:22.481548 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46132 (local address 127.11.102.60:42801)
0620 06:11:22.481670 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0620 06:11:22.481676 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 06:11:22.481984 (+ 308us) server_negotiation.cc:407] Connection header received
0620 06:11:22.482164 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:11:22.482168 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:11:22.482216 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:11:22.482258 (+ 42us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 06:11:22.482627 (+ 369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:11:22.483413 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:11:22.484663 (+ 1250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:11:22.484994 (+ 331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:11:22.485110 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:11:22.485451 (+ 341us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 06:11:22.485530 (+ 79us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 06:11:22.485595 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0620 06:11:22.485739 (+ 144us) server_negotiation.cc:299] Negotiation successful
0620 06:11:22.485790 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260620 06:11:22.486652 2200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:11:22.481438 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40318 (local address 127.11.102.62:39415)
0620 06:11:22.481829 (+ 391us) server_negotiation.cc:206] Beginning negotiation
0620 06:11:22.481833 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 06:11:22.481848 (+ 15us) server_negotiation.cc:407] Connection header received
0620 06:11:22.482177 (+ 329us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:11:22.482180 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:11:22.482359 (+ 179us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:11:22.482471 (+ 112us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 06:11:22.482848 (+ 377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:11:22.483694 (+ 846us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:11:22.484806 (+ 1112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:11:22.485161 (+ 355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:11:22.485266 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:11:22.485623 (+ 357us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 06:11:22.486007 (+ 384us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 06:11:22.486111 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0620 06:11:22.486433 (+ 322us) server_negotiation.cc:299] Negotiation successful
0620 06:11:22.486499 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":306,"thread_start_us":90,"threads_started":1}
I20260620 06:11:22.489913 11672 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9HMp43/build/debug/bin/kudu with pid 1838
I20260620 06:11:22.496436 11672 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9HMp43/build/debug/bin/kudu with pid 1913
I20260620 06:11:22.502786 11672 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9HMp43/build/debug/bin/kudu with pid 1989
2026-06-20T06:11:22Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (3164 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (6277 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (279377 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST